 So now it's time for power worker, sorry. So power is a release engineer at Samsung on the Tizen platform, which is a mobile phone. And Pavel will talk about testing remote embedded device. Mike is yours. Thank you. Hello, everyone. As I was introduced, I work at Samsung R&D Institute Poland, and I'm currently involved in Tizen Release Engineering team. That's why we have to automate as much as possible in order to stay efficient with our day-to-day activities. That's also why we came up with the design for unification for testing automation on embedded devices, which we call Max Pi. And that's what I would like to share with you today. Let me start with a short introduction to what I will be actually talking about. Then I will summarize previous efforts in this matter. I will depict our idea for testing automation on embedded devices. Then I'll go through the hardware that we made, software, accompanying software as well. And I would like to share with you our future plans for the work that we still have to do. I will summarize it with a short conclusion, and then we'll have a Q&A session. So what do I actually do? I work with Tizen, which is GNU-Linux distribution for embedded devices. It is based on standard components, like GNU Tools collection, Linux kernel, obviously, Wayland server, Wayland display server, and Enlightenment Foundation libraries, or EFL in short, as the main graphics library. And Tizen is mainly used on consumer devices, like mobile phones, wearables as well, but also TVs, fridges, as in Family Hub. And since May 2017, with Tizen 4.0 milestone one, on IoT devices as well. But Tizen on those devices is a product one, great Tizen. What I do is work on platform, which is being developed by contributors all over the world with R&D centers in India, USA, South Korea, and Poland as well. That's why we need to develop our platform continuously with daily snapshots of merged software. But the fact that we release often in order to fail early does not mean that we take the QA stage lightly. Because we have to take care of devices for developers all over the world, we always check the software that we are to publish on actual devices before the images are published for everyone to use. And even though packages are equipped with internal tests and we run them on the daily basis, they are not enough. Because we want to know unexpected behavior of developers' devices, that's why we make sure that there is no. And how does it look like? Well, we monitor pre-releases of the software, download them, flash onto the target devices, then run test batteries, gather results, and then investigate if there were any failures or other indicators that the software should be stopped from being published. But in order to do that, it is much easier not to just have all of those devices, various devices, mobiles, wearables, or even fridges right on the desk of release engineers. It is much easier to have them being accessible remotely. It is especially important for the devices that are not published yet, that they really need to be start in a secure manner. But it also gives us the ability to have much less effort on the developer's side as for the maintenance of the devices, and all of them can be much better utilized when they are shared, when they are never pointlessly stored in some developer's locker once he or she goes back home. That's why we came up with this device. It's MaxPy, and it needs only two interfaces, two connections, in order to be accessible remotely, which are power supply, of course, and the Ethernet for network connection. Once connected to the target device such as Raspberry Pi 3 right here, it is already ready to use, and it's, as you might suspect, pretty easy to scale because it needs only two interfaces. So scaling up is as easy as connecting further MaxPy's with target devices to the network switch, and it unifies the access to the boards. So it's easy for Raspberry Pi's or other single board computers, such as Odroid, or even other types of target devices like mobile phones. And we will get back to the MaxPy's in a minute, but before, let me summarize real quick why did we came up with this design and what else can be used for automated testing. From software point of view, one of the most popular applications for that use is Lava, which stands for Linaro Automated Validation Architecture and is being developed by Linaro. It is an automation system for operating system deployments and supports both virtual and physical devices, and allows running bootloader and system level tests as well, although some additional hardware might be necessary for them to work properly. However, what was a problem for us was being able to run interactive sessions on boards that were connected to the Lava server. Lava has internal mechanism for interactive sessions, which is called hacking sessions, but it did not work for our use cases, which involved modification of bootloader or rerunning or restarting device over and over again. For us, it was to test recovery mechanism of Tizen platform. However, it is widely used at Linaro, of course, for automotive-grade Linux test lab system and for other popular GNU Linux distributions, such as Debian or Yocto as well. I believe the most popular use case is kernel CI with already three and a half million successful boots and counting, and that's not the only effort from Linaro to automate testing on embedded devices. In 2013, they also came up with LMPs, which are boards for both testing hot-plugging devices and checking how the software on embedded device react to hot-plugging, for example, SATA device or USB device, but also they were able to give much better control to the administrator of testing laboratory over the device under tests or that for short, but it is currently on hold and lessons learned from use of LMPs, especially of microSD card demultiplexer were published in the link down below. However, we also gave an attempt at creating microSD card demultiplexer or SD Max for short, and we've been using it since 2016. This board only gives you the ability to share access to the microSD card between device under tests or that for short and test server or TS. It also allows to control the power supply to the target device, and that's basically it. It does not know anything else about target device, and it only gives you these two small features. However, it was pretty useful for us and not only for us, Qt company and also Ableton, which you might have heard of, were able to recreate this setup because we thought it would be great to share it with others, and we published all these schematics and software and other related materials in the Git repository and use only open source software for that so that everyone can use it or even fork it, like Reslin.io, who thought that it would be great to have some indicators on board and created AutoHat board on the basis of our SD Max. Unfortunately, we ran into some issues using SD Max and the main issue that we got was the instability of the connection on the USB between the SD Max and test server. Every three to six months, the test server was unable to enumerate another devices on the USB hubs and that the 71 errors are protocol errors, which led to down times of our testing laboratory, which we were not all right with. That's why we took a step back, went back to drawing board and decided to bring a new idea to the table. We already knew what was working right, we also knew what wasn't, and that's why we decided that we will keep only replaceable medias, which were of the quickest in testing laboratory, so we still are using microSD cards for the deployments of operating system images, but we decided that there shouldn't be a situation where a failure of a single device causes a failure of the next ones. So we tried not to introduce any SPOF onto the setup and we also knew that there has to be no involvement of the USB connections on the test server side. We also wanted minimum external connections between the device under target or the board that controls it outside world and wanted to have it in a unified way. With the feedback we got from ResinIO, we decided that easy setup and maintenance would be a great plus and since we were back at the drawing board, we thought that maybe a whole user interface would be helpful and with increasing demand for measuring and lowering of course power consumption, we decided that it could be useful as well as having and as being able to write some EDID information to HDMI. It's pretty useful for the devices that you would like for them to think that they have display connected even though they don't. That's why we came up with this design but I think it would be easier to describe it on the schematics. We started with the Gold Alt SD Max which has already a simple UI, edit to the board itself and of course the power control and power consumption measurement on board as well. Everything is under control by the Nanopinio board which is a single board computer ARMV7 and of course a microcontroller for low level features of the board itself. With the MaxPy board, we are able to demultiplex the connection of microSD card between DAPT and TS. We are also able to switch power supply and that's what SD Max already gave us. We are also able to switch dynamically jumpers on the DAPT which is pretty useful if your device has some buttons which are required to be pressed in order for that to boot up or to be put into the download state for flashing device or so. We also have got hardware for measuring power consumption and the ability to write fake ID to the device under test over HDMI connection. We also provide multiple ways of connecting to the device under test via USB, Ethernet, microSD card of course or if you need a serial console directly to your device under test that's also available as well. We wanted the MaxPy's to be able to interact with testing laboratory maintainer. That's why it is equipped not only with activity LEDs such as SD Reader LED or Power LED which were suggested by ResinIO but also RGB LEDs on the bottom left corner for indicating state of the device not only MaxPy but also device under test and also OLED display for simple messages. We also put a couple of buttons to be able to interact quickly with the whole setup. It gave us easy maintenance over the... over the dryad which we call MaxPy board connected with target device but we knew that not everything can be predicted before those boards go into production. That's why we decided to be able to extend them from start and that's why we use the prototype boards for quick setups if there is a need for a new device or mode or just rearrange something quickly. With all that we've got finally an independent... independent controller for devices under tests which does not require any test server which is aware of its state and can indicate any failures or unexpected behaviors which is also easy to maintain and extensible from the ground up if there is a need to be extended and if you'd like your own you'll have to equip yourself with a NanoPy Neo for controlling it for about 10 bucks with pads for the MaxPy board and the $80 price was initial batch which was about 30 boards but also some high soldering skills and a lot of patience but if you're still interested go ahead to get TizenOrc to our repository and to check the SCAM attacks out but hardware is not everything that had to be provided in order to have the automating testing laboratories for embedded devices we also had to implement some accompanying software and with previous efforts I mentioned for example Lava which is Monolith we decided that it is hard to extend it in future so multi-tier architecture was chosen for the software to control test farm we also wanted to follow the Unix philosophy and each tool does only one thing but we hope that it does it well we wanted them to be able to communicate and to be easily swapped if there is a need to provide a new tool so they all communicate over HTTP APIs but still we wanted to have a simple and similar solution stack and everything is written in Go language and as I mentioned earlier how does the testing look like that's why we provided for layers for testing laboratory I mentioned that we have to monitor and download all of the software to be tested and who knows how to do that that's the role for Perun Perun is a character from Slavic Legends it's demigods who has the most power and most responsibilities of all others and who knows which actions are necessary that's the work of Velez Velez in Slavic Legends is the one who takes care of the underworld as for the tasks where can it be done that's the work for Boruta Boruta is a caretaker in Slavic Legends and takes care of all living creatures in forests and as for how to actually do that that's the work on hardware level and actually MoxPy which does not follow the Slavic Legends because it was invented independently much earlier and let's begin with MoxPy which actually allows us to create automatic testing laboratories it manages single device under tests and is fully aware of its capabilities whether it has display connected or not which interfaces does it provide and so on and so forth it requires only two interfaces as I mentioned before which are power supply and the Ethernet for network connection together with device under tests we call it Dryad which also is from Slavic Legends as for the software to manage Dryads we provide two tools first one to be able to put operating system images or flush over the air to the devices and also to the other one to control power supply to the target device to put the device into known state and be able to control it and with all those over all those Dryads we've got Boruta which takes care of the whole farm of multiple Dryads it can schedule access requests to the devices and it provides a convenient way of accessing selected Dryads one it is given single Boruta server takes care of multiple Dryads and a single Dryad hooked up into the Boruta server starts in maintenance mode which is then moved to the idle or allocated by the testing laboratory maintainer if such Dryad matches requirements for the access requests then Boruta prepares environment for that set ups tunnel or SSH tunnel or other way of connecting to the device under test if it is necessary and then actions can be performed on the target devices directly once they are finished devices go back into the unallocated state and can be reused after that but I believe no one wants to do that on their own all the time that's why we also provided a VLS which is a lightweight testing framework inspired by Lava and it provides similar to Lava interface based on YAML job definitions which we try to stay as close as possible to the Lava syntax so we also allow the same stages of device under tests which are deployment both tests and of course collection of the results single VLS server should be connected to the Boruta and then Boruta who controls Dryads can execute all those tests and with such setup we've got it's pretty similar to the one that Lava provides because from VLS YAML job definitions goes in then they're all executed right down to the target devices and the gathered results are brought back by the VLS web interface and that's actually its main purpose to automate the actions that are being performed on the device under tests and actually that's all that VLS provides it only parses the YAML job definition collects all the assets that are necessary for the job to be completed requests access to the device under tests from Boruta and then perform the actual actions but I also mentioned that there should be a layer which actually has to know what should be done and that's the work for Perun which is our layer for or maybe a dashboard would be a better word for actually testing the operating system images which schedules the verification on the basis of what is being pre-published for Tizen Perun is the layer that actually automates the quality assurance step of what we do at Tizen every day so the release engineering duties Perun also has to be connected directly to the VLS server and that's how Perun provides us with results it crawls a single URL with operating system images reports any change that could be detected submits VLS jobs on those detected changes collect artifacts from the test execution interprets them so that we won't have to and that's how we know whether a new release can be published or not and with this layered setup we think that it's kept pretty simple but also decoupled so you only need software and hardware of course to unify access to your various target devices then go ahead just take maxpice, connect them to your devices and with two interfaces power supply and network connection you'll have access to them remotely if you already have such hardware and you don't want our then replace it with your own but maybe you'd like to be able to schedule your tasks or schedule remote access requests and for that you might find our Boruta useful or maybe you already have your own testing framework you do not want VLS but Boruta with maxpice would be pretty useful or maybe you think that our Golang implementation should be reworked and you have already your javascript dashboard your Python based testing framework and you schedule access to the devices under tests with Java Jenkins but maybe the maxpy would be the hardware you would like to use for the unification of your environment that is also possible as long as the communication of our HTTB APIs is set with the specification we provided so that's what we've been working so far and what about the future we will focus on the audio I.O. and testing the audio functions of the single board computers and other embedded devices as well we will investigate the possibility of having USB type C as more and more devices are equipped with it and it could be pretty useful and for the better control over the dryads we also think about having serial console with embedded converter on the board itself as for the software side we will provide convenient web interfaces for current layers since what we currently have are only CLI tools we also want to provide service state management in case there are some failures on the server side and also provide a new layer or even more of them to automate the whole setup further and if you'd like to know more about those devices go ahead to our wiki pages or where you might also find our lessons learned from the SDMAX board if you have some questions go ahead and drop us a line at our mailing list or podcast at IRC channel so to sum it all up we currently have got a setup which can be quickly made and is pretty easy to maintain we finally divided all the responsibilities that are in testing laboratory and no developer has to do them all by himself we've got parallel execution of all those tests on target devices and sharing devices for better utilization and unified environment which we've been pretty happy with so far if you have any questions about this board or software or the laboratory setup I would be now more than happy to answer them the question is how many devices do we have in our testing laboratory and currently I believe it's around 50 the next batch and around 100 but we never went with more so far the question is about Lava Dispatcher and running it on the Max Pi board we haven't tried it but it would be possible if you ran Debian on the Nanopi board and got this directly from Lava Maintainers because it's already prepackaged in the repository if I'm not mistaken so yeah that would be possible we just haven't tried it yet we focused on the Boruta Veles Perunstak as far as I know the question was about the maximum power that can be provided to the target device and as far as I know there is no limit but I believe that it would be much safer to ask that on the mailing list directly to the board designer alright so the question was about making a bigger batch of Max Pi boards and about gathering all interested on the Tizen mailing list Tizen is an open source project so everyone is invited to discuss and to post new information there so I see nothing that could prevent from it so the question was about the HDMI on the board and getting image from the HDMI currently only the PCC line is connected for the idea of the information and it's having the image over the HDMI is a complex topic and for now it's out of our current scope sorry