 Are we going to do some testing? And what about this kind of activities? Tell us more about it. Hello, everyone. My name is Paweł Wyczorek. I work at Samsung Currently Institute Poland, and I am currently involved in Tizen Release Engineering team. Releasing operating system images, as you might imagine, requires a lot of validation, verification, and in order to stay efficient at these day-to-day activities, we try to automate as much as possible from our daily tasks today. I would like to share with you our recent design in a topic of embedded Linux testing, which is MagSpy. Let me start with a short introduction to what we'll actually be talking about today. Describe in short previous attempts and efforts in this matter, present to you the idea behind it, and of course both hardware and software connected with it. Then I will share with you our future plans and summarize it, and we'll have a short Q&A session afterwards. So what actually Tizen is I mentioned earlier. Tizen is a GNULINUX distribution aimed at embedded devices, mainly mobile phones, wearables, but also TVs and even fridges, or IoT devices with Tizen 4.0 Milestone 1 in May 2017. But before, Tizen goes to products, such as these examples of consumer electronics. We develop a platform, and Tizen as a platform is developed by various R&D centers all over the world, in India, South Korea, USA, and Poland as well. That's why we have to do it in a continuous way, with at least a single snapshot a day, but often even more. That does not mean by no means that the Q&A stage of all those OS images is taken lightly. We take a great benefit from internal tests provided with software packages that we release, but they are unfortunately not enough. We care about the devices that are used day-to-day by platform developers, and do not want any unexpected behavior taken by a new update that was released without being tested on an actual hardware. But testing new releases on various devices I mentioned earlier is not an easy one if you have different devices in different parts of the world. That's why it's much easier to be able to access them remotely. First of all, it allows us to store them in a secure manner without any possibilities of security leaks during mailing all those devices. And also it is much less for developers if they do not have to maintain all those devices by themselves. Also, they are much better utilized if they are shared and not stored in Locker after the 5 p.m. once everyone goes out of office. With all that in mind, we decided to create those, but before it was produced, there were some previous efforts. The most notable one is Lava, linear automated validation architecture, which aims to automate deployments of operating system images on both virtual and physical devices and allows running wide range of tests on those devices. However, there are some use cases. For example, ours were having interactive access to the device under test that are not yet very well supported in Lava. Although there are attempts at doing so, like built-in hacker session mechanism or the Lava Ball project by Free Electrons. Still, Lava is widely used, for example, for automotive grade Linux, for Debian, for Leonardo as well, of course, but I believe the most popular use case is Kernel CI, where already we have over 3.5 million successful boots and counting. As for the hardware attempts in automating testing of embedded devices, Lava, linear also prepared LMPs in 2013, which aimed at checking behavior of embedded systems during hot-plugging, various devices, for example, SATA devices, USB devices, but also providing better control over the devices under tests, like with microSD card de-multiplexer, which can share the access to microSD card between test server and device under tests. Unfortunately, those efforts were put on hold and lessons learned from that were described in the link down below. We thought that we could give another try to those efforts and came up with our microSD card de-multiplexer in 2016 and used it ever since, which of course shared access to microSD card and allowed controlling the power supply to the device under tests and that was pretty much it. It did not preserve state of device under test or anything. It only provided those two interfaces. We thought it could be useful. It was useful for us, for our time being and we shared it on the git linked down below. As far as we know, Qt Company and Ableton, which you might have heard of, were successful at creating their own SD-moxes and plugging them into their own testing laboratories. But these were direct clones. Some thought that this design could be improved, like resin IO, whose engineers forked SD-max and came up with auto-hardboard. Feedback we've got from resin IO mainly focused on being able to know the state of the device under test during the actual test procedure. Auto-hardboard is equipped with multiple state indicators. But as you might have already suspected, SD-max had some issues, mainly because of the connection between the controlling board and test server, which scheduled all the tests. The USB connection that we used was not reliable enough and from time to time it caused multiple failures or on all tested devices. We thought that it was the time that we should take a step back, go back to the drawing board and came up with a new idea. We already knew what was working, what was not, so we decided to stay with only replaceable media because the storage devices were off most quickly and decided to drop all of the USB connections between test server and the device and the controlling board, of course. What's more, we wanted to have as little external interfaces connecting the controlling board and the external world as possible and wanted also unified access to the devices under tests. That's why we designed the Max-Pi board, which actually consists of multiple designs that were already in use. For example, well, as I mentioned, it is equipped with only two external interfaces, so power supply and Ethernet connection for being accessible from network and the good old microSD card the multiplexer that we used already before. But that's not everything. Following the feedback from ResinIO we also added some UI for the convenience of testing laboratory administrators and power control and power consumption measurement hardware directly on board. Everything is being under control by the small single board computer, Nano-Pi Neo, RMV7 based and of course microcontroller for low-level functions of Max-Pi board. And what about the external connections from the controlling board to the device under tests? Well, the most popular ones and probably best supported on various embedded targets are Ethernet and multiple USBs including the USB with the 5th pin or the ID pin. If that's not enough, if that is not your use case, or you require some being able to debug your board even further, you also got a serial console between the controlling board and the device under tests. And if your boot procedure or putting your device into the download mode for flashing requires pressing some buttons, you can simulate it with dynamic jumpers that are already on the Max-Pi board. We also equipped it with HDMI but this HDMI connection does not provide actual data from the display. It has only PCC line connected for sending EDID information so that the device under test might think that it has a display connected. It is pretty useful in verifying the on-screen data. But we also knew that not everything can be predicted and that's why additional extensions are also supported on the Max-Pi board as well. But let's get back to the feedback we've got. And for the convenience of the testing laboratory administrators we put the Activity LEDs on board suggested by the AutoHat board design. So we've got the SD Reader and PowerLED but that's not all. We like flash of things so we put also double RGB LEDs for indicating state of both Max-Pi board and device under test and for presenting short messages we also got an OLED display. If there is a need for quick interaction with the Max-Pi board administrators can do that with the buttons as well. So we've got it pretty easy to maintain if there are multiple Max-Pi boards connected to the devices but the extensibility also came in handy already a few times. Together with the designs of the Max-Pi boards we will publish the designs for the prototype board which you might be able to extend its abilities even further if there is a need for that. So with all of that we finally got an independent controller in the testing laboratory in contrast to the SD Max which required a testing server. We finally have got the device that is fully aware of its state and of the state of the device that it controls which is easy to maintain and indicates any failures that might occur during verification of operating system images and is extensible. It's not a close design that we will not be able to change after it is already produced and if you would like to produce your own well you probably should start with Nanopainio which controls everything and costs around $10. The parts that were used to create Max-Pi board cost around $80 but that differs depending on the amount of the boards that you want to produce. The $80 price was for around 30 prototypes that we got at first and of course it will require a lot of high soldering skills and a lot of patience but if that does not scare you off go ahead over to the Git repository and you'll find the schematics for the board but the board is not everything it also requires some accompanying software since the solutions so far were pretty monolithic and they did not fully support exchanging some parts with another we decided to go with multi-tire architecture we also wanted to follow the Unix philosophy of doing one thing but doing it pretty well so at least and still we wanted them to be able I mean those parts on the architecture to be easily replaced so the communication between them is made with HTTP APIs even though we wanted to stay the whole stack pretty similar so each layer is written in Golang and what are the actual layers since the task of verification OS images requires detecting changes in the pre-releases downloading all those images flashing devices actually perform tests then gathering results and interpreting them well for all those tasks we've got four layers first one which will do the job of knowing what actually requires verification which is Perun Perun is a character from Slavic legends he's a demigod who has the most power and the most knowledge and who would know which actions are necessary that's the job for Veles Veles is also a character from Slavic legends and he takes care over the afterworld who knows where all those tasks can be done that's the work for Boruta Boruta is a caretaker in Slavic legends and makes sure all living creatures are safe and sound and who would actually know how to perform all those actions well that's the work for Moogspy and Moogspy does not follow the the naming scheme since it was developed independently and let's start with that Moogspy manages a single device under test and has the full control over it it is also fully aware of its capabilities whether it has a display attached what kind of communication interfaces does it provide but requires only two interfaces to the external world to be able to access the device under test remotely which are of course power supply and network connection together Moogspy together with device under tests we call it Dryout and with multiple Dryouts well Boruta takes care of them Boruta provides convenient access to the selected Dryouts by scheduling access requests that are sent to the Boruta server over Boruta server we've got Veles which is lightweight testing framework and provides lava-like interface that are YAML-based job definitions that contain actions that should be performed on the device that can be accessed remotely from Boruta and over Veles we've got finally Perron which is a dashboard which actually performs the QA step of release engineering duties and allows us to automate our work we tried to keep it layered and simple and of course the coupled as well so if you would like to use only the hardware layer which are Max-Pyboards go ahead and produce for you if you already have your hardware in place but you would like to be able to schedule access and take care of multiple ones you might use only a Boruta server but if you would like to apply your own testing framework it might be useful for you to use both Boruta with multiple Dryouts you might also find our whole stack you might need some replacement over the whole stack so if you've got already your dashboard written in JavaScript and your testing framework in Python and you already schedule access to your devices under tests with Jenkins written in Java but you need only the hardware layer with Max-Pyce go ahead do so as long as the HTTP requests are correct and that's what we've been working so far and what about the future work we will focus on providing the audio I.O. for extending test cases and we will investigate the possibility of having USB Type-C on board since more and more devices are equipped with it and probably we are discussing that if there is a need to provide a serial console not only between device under tests and Max-Pyce but also to the Nanopyce board itself from the software viewpoint we will provide web interfaces for current layers since only CLI tools are provided so far and also provide better management of our service states in case of failure layers to automate release engineering work even further so to sum it all up we finally got both hardware and software a stack that is quickly that can be quickly set up and can be maintained pretty easily we divided all the responsibilities and taken a few of them from developers we also got better utilization of all devices under tests and finally we've got a unified environment if you have some questions you might find answers on our wiki pages but if there are further ones go ahead and drop us a line on mailing list or poke us at Tizen channel on free note I believe we might be running out of time so if you've got any questions I will be outside and will be more than happy to answer them thanks for your attention