 Hi, my name is Paweł Wieczorek, I work at Samsung Currenty Institute Poland and I am currently involved in Tizen Release Engineering Team. This, as you might imagine, requires a lot of verification, validation and in order to stay efficient with our daily tasks, we try to automate as much as possible from what we do daily. Dzisiaj, chciałbym share with you our most recent design for testing automation on the embedded devices, which is Max Byboard and Accompanying software. Let me start with a short introduction to what I will be talking today. Summarize real quick previous efforts in this matter, then I would like to describe our concept and show you the solution. I will move on to the accompanying software and present to you our plans for the future. Then I will sum it all up and we should still have some time for Q&A session. So to begin with, what Tizen actually is. It's GNU Linux distribution aimed at embedded devices, which uses standard components such as a collection of GNU tools, Linux kernel, obviously, Wayland server, display server, and Enlightment Foundation libraries, or EFL for short, as main graphics library. It runs on various embedded devices such as mobile phones, wearables, TVs, or even fridges known as family hubs. And since May 2017 also IoT devices, since the Tizen 4.0 milestone one release. But before Tizen goes to product, a platform has to be developed and it involves engineers from all over the world, from R&D centers in USA, South Korea, India, and Poland as well. And that's why we need to release often to fail early and to detect any defects that might come with the most recent patches. That's why we publish snapshots daily, but by no means it should affect quality of the software that is being published. So release engineers act upon it and there is a QA step prior publishing each snapshot. Although internal package tests are really helpful, it's not enough for us. And since we have to take care of devices that are being used by various developers, we tend to check all the snapshots before publishing them on actual hardware, so that we know that there will be no unexpected behavior or any damages made to those devices. As you might imagine, this requires a lot of devices and not all of them can be easily obtained, such as hardware that is not yet publicly available or freezes, I mentioned earlier. That's why we wanted to be able to access them remotely. This way there is no possibility of some security leaks. Developers can focus on software development and not on board maintenance, and boards are not stored aimlessly in lockers once developers leave office. But they can be better utilized by other developers from different R&D institutes. That's why we came up with the MaxPy board. And once it is connected to the device under test or that, we call it Dryad. And as you can see, Dryad requires only two connections to be able to be accessed from the outside world, which is power supply, of course, and network connection for communication. This Dryad is connected to the Raspberry Pi 3 board, and all other connectors are only between the MaxPy and Raspberry Pi that consists of this Dryad. Since Dryads require only two connections, it is pretty easy to scale it up, and it is as easy as connecting multiple Dryads to the network switch and providing a power supply for them. It also unifies the access to all of these devices. It does not matter if it is Raspberry Pi, Exynos-based Odroid, or a mobile phone such as this Note 4, I believe. And we will get back to the MaxPy board in a second. But first, let me tell you which other efforts in this matter have been made so far. I'm starting with software approach. Most notable software is, I believe, Lava, which stands for Linear Automated Validation Architecture, and it is a deployment system for embedded. It supports both virtual and physical devices and allows running wide range of tests from boot and boot loader to system level, although some additional hardware might be required for that. It is used by various distributions like Automative Grade Linux, but I believe even more popular user of Lava is KernelCI with three and a half million successful boots and counting. For us, however, Lava was not a sufficient solution. We required remote access, I mentioned earlier, and remote access on a level that it acts as if the board was already on release engineer's desks, not the one that Lava supports via hacking session, which opens the tunnel to already booted up device under test, and then gives the access. As for the hardware approaches, Linaro also came up with LMPs around 2013. LMPs were boards that helped with checking hotplugging of various devices, such as SATA, USB, HDMI, and also they gave better control over adults. However, where their development was put on hold and lessons learned from development of LMPs is linked down below. Slides are already online. We also gave a try at hardware attempt at solving this problem and then came up with our SD Max board. This SD Max board requires connection to the test server. It is not an independent board and allows doing three things with device under test. It cuts power supply so that the dot can be brought into known state. What I mean is to be completely shut off. It gives pass through to the USB on the dot and of course it shares connection to the microSD card between the test server and device under test. Since it worked pretty well for us in 2016, we decided to make it open hardware and all of the schematics are available at Git repository and it has already been reused by Qt company, by Ableton as well and also forked by Resinio, who came up with the AutoHot board, which is SD Max with a few indicators, such as microSD card activity or power supply activity. But the SD Max boards also gave us a lot of headache. For example, the USB connection between the test server and SD Max board was not reliable one and every three to six months we got these protocol errors you see here and so far we did not detect the root of the problem. So we decided to take a step back, get back to the drawing board and came up with a new idea. With the experience from SD Max boards we already knew what was working well, what was not and decided to focus on replaceable media only, since they are the components that were off most quickly. And still you might want to have the actual media that you will run the software on and not just a network boot or similar. For example, for storage benchmarking or so. We also decided that there should be no single point of failure and what I mean by that is that a failure of a single subsystem, such as USB on test host, should not cause failure on more than one device. And also we decided that there should be no USB involvement from the test server with the problems of the SD Max. We also wanted to have absolutely minimum of external connections and decided to give two of them. We wanted also unified access, no matter what device under test is actually connected and with the feedback from ResinIO decided that the setup and maintenance of all the boards should be as easy as possible. So we thought that user interface might also be beneficial in such board with interest in power consumption measurements and initiatives such as PowerCI. We decided that power consumption measurement hardware could also be beneficial on board and also we wanted to be able to write some additional information to the device under test such as EDID via HDMI connection for supplying information on what hardware can be connected to the device under test. With all that in mind, we came up with the Max by board I mentioned earlier, but I believe it would be much easier to describe it on these blocks. So as I mentioned, only two connections to the outside world and we began with the good old SD Max. So the capability of the multiplexing access to the micro SD card between test server which in our case is this nano by board and ability to switch off the power to the device under test. Next, the UI often requested and often came up with in the feedback we got from the SD Max development. Also power control and power consumption measurement hardware is already on the Max by board and to control everything we've got the nano by Neo which is pretty small Arm V7 single board computer and micro controller on the board itself for low level functions. As for the connections to the device under test we've got Ethernet, various USBs including 5 pin USB with the ID pin and USB OTG if there is a need to have control over the UART that's also available for diapers which stands for dynamic jumper to be able to simulate pushing buttons or switching on off various jumpers. Some of the boards we test on require such actions in order to be put into the flashing mode or to be booted up such as minoboard it requires pushing the power button. We also have got the HDMI but as I mentioned earlier only PCC line is connected for writing the IDID information and we also knew that not everything can be predicted a priori so there is also connection for the add-on boards with that we've got the ability of switching micro SD cards between device under test and test server which in our case is this nano by board we are able to put the device under test into known state we can press buttons on it we are able to measure power consumption write the ID information and also provide various connections to the device under test over the Ethernet that is connected to the Max by board and we also are able to interact with farm maintainers able to interact with those devices and since we are at interaction Max by boards are equipped not only with those LED's Activity LED's I mentioned from the feedback from resin.io but also a couple of buttons for quick actions some additional LED's for indicating state of the either Max by board itself or the device under test that is connected to it and also all LED display for quick information which has been pretty useful so far as for the extensibility of the board we use prototype boards if there is a need to extend features on the Max by board even further or if there was something we did not think of while creating this board with that we've got finally a standalone device that does not need access to the test server anymore which is aware of its state and not only its state but also device under test that it controls which is pretty easy to maintain extensibility from the ground up and if you would like to build your own if it would be beneficial for you you should get equipped with Nanopineo which goes for around 10 bucks parts for the Max by board cost around 80 dollars and it was for I believe 30 units we made around December but also high soldering skills and a lot of patience still if you're interested go ahead to our git repository where you can find schematics software, schematics for the prototype boards and even more but that's not the whole solution hardware is not everything we also had to develop some software that accompanies this solution and with approaches such as lava, monolithic approaches we already knew that making changes after time can be painful so we decided to go with multi-tier architecture and focus on following the unique philosophy of doing one thing but doing hopefully well so we've got four layers in our software stack and those layers communicate with each other via restful HTTP APIs still we wanted our stack to be similar and to be able to develop each of the components by the same team so everything is written in Go language and as for the software layers there are four types of actions that have to be done as day-to-day release engineering activities and it is monitoring build system for new releases actually pre-releases since they are not publicly available yet or getting notification about them depending on what your build system supports also being able to schedule all the actions having access to the device under test and actually running those actions so the first one is being taken care of Perun Perun in Slavic legends is a demigod with biggest powers so he was chosen for the first layer as for the actions that are necessary to be taken care of it's the job for Veles in Slavic legends is god of the underworld and takes care of it as for being able to know where those actions can be done it's a job for Boruta Boruta in Slavic legends is caretaker of the forest and all living creatures and the knowledge as how to perform all those actions is a job for Dryads Dryads are the free souls of the forest and let's begin with those Maxpire based Dryads Dryads in our testing farm take care of a single device under test and are fully aware of all of capabilities that given that provides as I mentioned they require only two interfaces which are power supply and network connection and are equipped with software for flashing firmware over the air and also for cutting power connection or bringing it up again but that's just for a single device if there are multiple ones they have to be taken care of and as for the farm management system we've got Boruta Boruta schedules access requests to the devices and provides convenient access to the assigned ones so how does Dryad Lifecycle goes in Boruta first Dryad is in maintenance mode and it can be brought into idle or unallocated state by the farm maintainer if there is an access request to Dryad with capabilities which given Dryad serves and it matches then it is assigned the test environment is prepared SSH tunnel for the access to the Nanopie board is set up and then all of the actions might be performed so to sum it up it does not matter who requests the access to the Dryad if there is one that matches requirements it will be assigned to a requester and it can be either interactive session by real users or just an automated one by some automated testing framework and since we are talking about automated testing framework let's move on to VLS which is our lava inspired testing framework which translates YAML job definitions like the ones used in lava into the actions that are being executed on dots Actions can be divided into several sections such as deploy, boot, test and collect we try to maintain compatibility with the lava job definitions but there are only a subset of actions that can be run in lava laboratories it is supported in VLS so far as you might have suspected VLS main purpose was to automate the step of performing actions on the device and that's what it comes down to just parsing YAML, collecting all the necessary assets for job execution then requesting access to the device under test in Boruta and actually performing test and collecting results Multiple VLS servers can use a single Boruta server since Boruta only schedules access requests but let's focus on just a single one and no matter who submits YAML job for the test execution or whether it's user or automated system it doesn't really matter for VLS VLS takes it from there and takes care of the whole testing process Moving on to the final layer of our new testing system, Perun which is the testing system for the OS images it schedules verification per new pre-release and actually automates the quality assurance steps of release engineering duties it crawls given URLs then reports any changes it detects submits, VLS jobs collects all the artifacts after the testing jobs are completed and interprets the results so that we know whether new pre-release can be accepted or should be stopped from being published Just like with VLS there can be multiple connections from the Perun servers to VLS for example to monitor various build systems which can publish software to be tested on All those layers combined make the Slav stack which we named it and the name comes from all of those Slavic legends We wanted to keep it simple and the coupled so if you only need unified access to be able to access remotely your devices under test in a unified way go ahead, take the schematics for the Max Byboard and your all set but maybe you already have your own solution for the remote access to the dots but you still want software that manages a whole farm Replace Max By with your solution Take Buruta as long as the API's contracts are fine it should work properly Maybe you have already your own testing framework but still could use board farm management software then go ahead and take multiple parts such as Buruta and Dryads Or maybe you think that our Golang implementation requires reworking or you already have your javascript based dashboard Python testing framework and Jenkins for example for scheduling access to the boards As long as the API contracts are maintained all these layers can be easily swapped So that's what we've been working so far and as for the future plans on the hardware site we plan on having the audio input and output on being able to test audio IO and we plan to investigate the possibility of extending Max Byboards with USB Type-C since more and more targets are equipped with that connector Also we see that there is a need to be able to access serial console on the Nanopi boards and the future revisions of Max Pi will be equipped with it On the software side we will provide web interfaces for all current layers So far we only used CLI and it was enough We also intend to provide service state management for to be able to to work with failure if such occurs and provide further layers or maybe to automate release engineering duties even more Further details can be found on wiki pages either for Max Pi boards or lessons learned from our experience with SD Maxes If you would like to for us to help you with creating your own Max Pi please do not forget to visit our website www.summedle-up.com We finally got a setup which can be reused pretty quickly and is easy to maintain We finally got all the responsibilities divided so developers only on software development testers on writing tests and farm maintainers on maintaining all those devices We've got parallel execution no matter who wants to access the devices either automated testing systems or interactive sessions for real users We finally got the unified environment no matter which type of device is connected That's all I've got for you prepared today and if you have any questions I will be more than happy to answer them I will repeat those because I've got only one mic The question was about the fact that I only linked the Max Pi repository is because it's the only one that is already publicly available All other ones are still in the process of being published So Welles is halfway through Perron is still in development and Buruta should be published in a few weeks but so far there is no publicly available documentation If there are no more questions thanks for your attention and if you would like to see all of those in action I have to stand on technical showcase tomorrow evening if you'd like to see dryads or Welles, Perron only Welles and Buruta Perron not yet but it's coming and tomorrow night evening sorry, at technical showcase I will be demonstrating the software and hardware as well if you're interested For now, thanks for your attention and have a good one, thanks