 Hello, everyone. My name is Parowie Czorek. I work at Samsung Currenty Institute Poland. Currently, I'm involved in release engineering process for Tizen Common Linux distribution. The distribution is targeted at supporting as many different developer boards as possible. Dlatego, about a year ago, we started a project an automated testing laboratory to check all those images that we publish on boards we currently support. That's what I would like to tell you about today. I will start with a short introduction to our testing laboratory. Then I will present to you what drove us to creating such a solution. Następnie opowiadajmy, co musieliśmy zrobić i jak robiliśmy to. Następnie opowiadajmy się z twoimi planami na przyszłym razie, do końca. I opowiadajmy to, jeśli czas pozwala, to też będzie Q&A sesja na końcu. Tak więc, zacznijmy. To jest to, jak jedna noga na naszym automatycznym testowaniu laboratoryjnym wygląda. Jak widzicie, są tam mnóstwo grupy targów, a też jakieś podróże w architekturach. I będę całkowicie cieszył się z was. Mam nadzieję, że nasze testowe nogi na naszym testowaniu laboratoryjnym wygląda to całkowicie dobrze. Niestety, to jest to, jak wygląda, jak jest całkowicie podróże. I jest całkowicie operacyjne. I zapomnijmy, co mamy na porządku. Jak powiedziałem wcześniej, mamy grupy targów, mnóstwo grup, a pierwsza na lewskim jest porządka minoborów. To są architektury targów, które używamy, aby testować 32-64 bity targów. Następnie mamy 2 Odroids U3+, w centrum testowania na ARM V7. I też hi-keys, dla testowania na ARM 64-bit. Mamy też 2 regularne, u stoku, u razu USB Hubs. I są bardzo mięsko. I znalazłem, dlaczego, w latach. Mamy też na górę i na górę 2 rozwoju na górę customowe, które są MicroSD karty D-Multiplexers, które na górę widzą, mniej więcej, jak to. I ponieważ jestem całkowicie szczęśliwy z Tobą, te górki są już naszą drugą wizją MicroSD karty D-Multiplexers. Pierwsza górka była nie tak szczęśliwa, jak ta, która używaliśmy tutaj. Więc przeszliśmy do nowego górka w całej górce testowania. Ale najpierw, powiedzmy, co nas odwróciło do tworzenia takiej solucji. Jak są nowe zmiany, jak są nowe zmiany, które wytrzymały w TizenLinux distribution? Myślę, że większość z was jest bardzo familiarizowany z pracą na górę. I dla większości zewnętrznych pracującymi, a kiedy podróż jest skończona do reprezentacji gór. Z powrotem, że górka po prostu zaczęła skończyć, a to jednak wytrzymało wytrzymałego, aby skończyć z powrotem po prostu, żeby skończyć do oprogramowania podróży. To jest naprawdę prosty, ten gór, który dodał do obiektów gór. To po prostu, to jest naprawdę prosty, open-built-service-serwer, który nie tylko ulepsza pakieta, ale też wszystkie swoje opowiedzi, aby zrozumieć, że niektóre nowe ulepsze są zainteresowane. Kiedy to wszystko jest całkowicie nowe imię jest ulepsza, aby to mogło być sprawdzone i przytrafiło się w następnym rozleciem Tizencomonu. Ten pracownik jest przede wszystkim automatyczny, ale jest jeszcze jeden punkt, kiedy humana interakcja jest potrzebna. I to jest działanie rewelacyjnych. Rewelacyjne decyduje się, czy nowe zmiany powinni być przyzwyczajone, czy niebezpieczone. Jak widzicie, to jest działanie naszej prasie prasie, czyli open-build-service i Jenkins. To są jedyne, które najmocniej się zainteresowujemy. Niestety, nie są naprawdę podróżone, więc musieli być robione jakieś aktywy dzisiejsze przez wszystkich rewelacyjnych. I co robimy? Przede wszystkim, jeśli pewnego zainteresowania się zainteresowało, rewelacyjny jest jedynym, który musi zainteresować te możliwości. I również musimy sprawdzić, czy nowe zmiany nie zainteresowują rewelacyjne. Więc testujemy, czy nie były niebezpieczone zmiany w api, czy nie były niebezpieczone zainteresowania, czy na podstawie serwisów. Kiedy wszystkie te zmiany są zainteresowane, możemy bezpiecznie sprawdzić nowe zmiany na rewelacyjne. Niestety, to daje nam dużo głównym. To causes some problems, mainly because all of these activities are time-consuming. We have to download all of the images for our supported devices from main.tizen.org server, then flash the whole farm of target devices and run all the tests. And this task is not only time-consuming, but also monotonous. It involves replaying over and over the same set of actions, but still it requires precision and focus from the release engineers who perform those tasks. So we asked ourselves, maybe the images that are created should not be tested this frequently. Maybe only major releases can be tested so thoroughly. Or maybe just simple tests would be sufficient. Or maybe we could assume that if there were no built breaks, no regressions in the creation of RPM packages which are used by tizen distribution. Maybe it is enough indicator of the successful upcoming release. Unfortunately, that would violate all of our basic principles that we try to follow in the releases. First of all, we try to resolve all the issues as soon as they are discovered. They are much, much harder to find once we decide to check for them rarely. Also, we do not tend to look just for a quick workaround. We try to always introduce a solution to the problem. And the most important principle is that we don't publish images for developers if they were not tested on an actual device. All those images that are published on tizen.org are safe to be used on your target devices. That is why we decided that all of these activities should be automated. And that's what we've been working on. All of these tasks could be categorized in one of the five major groups. First of all, we've got the easiest task to do. So the software related, like polling OBS for new images or simply getting them to the target devices. Then we had to create infrastructure for our build farm to run all those images and tests on. Next, we also had to interface with some external infrastructure outside our laboratory on main tizen.org servers like publishing test results so that they can be reused, reproduced and easily available to anyone interested. And finally, probably the hardest one, was the hardware related group. And it was mainly finding a unified way of flashing target devices with new images. But let's start with the easy ones. The open build service server, as I mentioned before, is not really well suited to work with other parts of our infrastructure out of the box. It lacks any event mechanism in its default installation. Those can be used by some external additions to OBS, but require much, much configuration. Also, all the naming conventions used by OBS are designed to be easily readable by humans, by release engineers. So there is need for parsing all those human-readable names. Also, all the new images are scattered through tizen.org servers due to some remnants from the categorization of all those images. We decided that we'll work with the technology that we currently know best and all those scheduling tasks or all the queuing tasks are currently done by Jenkins server. We also experimented with some lighter alternatives like tasks puller or build bot. And these experiments are still ongoing. But this is what we've been working with for the major part of last year. The second problem that we had to deal with was establishing reliable communication with all devices in testing farm. The regular, the most common ones, like OpenSSH communication or serial console are great and they serve their purpose well, but they do have their drawbacks. OpenSSH depends on network services and we try to keep things as simple as possible. So we would like to detect a failure in any network connectivity before we even try to communicate with an actual device. Serial console, on the other hand, is a much less flexible solution and it offers a lower rate of data transfer. The default choice for communication with our target devices is a tool which is provided with Tizen developers package with Tizen SDK and it's a smart development bridge which is used for most of the devices available in our testing laboratory. It combines best of both worlds. It depends only on a single service. It is still flexible like the SSH connection and it provides us with decent file transfer rates. We also had to provide a way of creation or maintaining test servers and we started with providing just a simple handbook for newcomer to the testing laboratory. It was based on Python tool for documentation, it was based on Sphinx, but it soon was clear that it is not enough. The pace of the changes in test lab was too high for maintaining a separate handbook with all the changes, with all the recipes on how to create testing laboratory. That's why we decided to maintain a git repository with configuration for our hosts to our testing laboratory. We decided to use the probably most popular Python-based configuration management tool currently available and that's Ansible. This lab hosts contains Ansible playbooks that allow to quickly and easily set up whole infrastructure necessary to create our own private testing laboratory. This is why we... The main reason why we decided to do that was to improve deployments of new versions of our testing laboratory and to get rid of the servers that were custom designed that only a single person knew how they were configured. This is when we finally got rid of so-called Snowflakes. As for interfacing with external infrastructure and the main use case was to share and publish our test results, we knew that all the results that we get should be easily available to everyone with the possibility for future reuse if someone would like to reproduce them or check some historic event. We also didn't want to introduce a new service and that is why we decided to publish all the information that we gather on Tizen Arc wiki pages. Fortunately enough, Tizen Arc wiki pages are based on media wiki which is provided with really simple but still flexible tool for automated editing and gathering information and all of this is done by PiWikiBot. This way we can share the whole environment information with anyone interested. All the test results are published right away and they can be used by anyone who would like to check or read them. The final problem that we had to face were actually different ways of getting Tizen images onto the target devices. Most of them required completely different procedures for getting the images onto them. All the procedures were designed mainly for a single device per host, not for a whole built farm. So there were common conflicts if there were too many target devices connected to a single host. Also, most of the procedures were architecture specific. But we found command denominator on all the target devices that we currently support. All of them are bootable from microSD cards and that's what we decided to use. That's when we came up with the design of microSD card de-multiplexer. These boards provide shared access to the microSD card between testing host and the device under test. Two parts are intentionally made easily swappable from the board and are not on the board from the start. These are actual memories and microSD card readers. Those two parts are the quickest to deteriorate, the quickest to wear off. That's why they should be easily and quickly swappable. The boards were kept as simple as possible. So the minimal set of connections is provided on each of them. First of all, the board control. Secondly, but not less importantly, slot for memory card. Also connections to the control over the target device itself and the possibility to get the microSD card connected to target device and the corresponding slots on the host side. I to have the authority to reset the device we've got also a power switch which is a simple relay switch for cutting the power to the target device. All the boards come with a simple tool which allows us to have the full control over the board from getting the information through controlling the naming on the board up to providing all the capabilities of the boards itself. With this solution we could finally close our former workflow which still involved human interaction to the fully automated one with the release engineer's work replaced with the simple piece of silicon. A since the boards saved us a lot of work we decided that maybe they could save a lot of your work as well. That's why we decided to publish all the schematics both for the boards and for all of the connectors that allow to interface with target devices and all of the schematics and sources are available at tizen.org-git repositories and were created using only open source tools so that they are easily available to anyone interested. As for the to-do list that we still have ongoing we currently are in the process of creating pretest cases so that we could detect failing images as soon as possible and do not waste time on the changes that will probably not be included in future releases. We also have to monitor changes between subsequent images in a more detailed way. Currently these are only changes in SH1 sums from the Git repository. Hopefully some method of bisecting all the changes will be introduced soon as well. We would like to be still able to get at least partial information from the failed test runs as well and we intend to improve our resource management on the whole build farm also by distributing the whole setup so that it won't be bound to a single location at Samsung R&D. I would like to share with you three main lessons that we got from creating the whole automated testing laboratory. First of all, modern automation does not need to reinvent the wheel. All the building blocks or the tools that serve automating the work are already there. They just require a little configuration from our side. Secondly, although this task might for some people seem intimidating at first, designing and producing custom hardware simplifies most of the tasks and allows to make the whole process automatic. And finally, automation, even though it requires some initial cost, pays off in a long term like any process that is given only to the server that works for us. There are some questions from you. I will happily answer them. The question was about the framework for the tests. I repeated your question, but if someone would like, the microphone is in the center of the room. Currently, we use three. The first one are the simple shell scripts that were used formerly by release engineers. These are the ones that we are familiar with and the test results that we are able to quickly scan through. Secondly, we wanted to unify the form of the tests. So we migrate most of our tests to Avocado Framework, formerly known as Auto Tests, if I recall correctly. There is also a third one, but it's connected mostly to Tizen Linux distribution. It's TCT, which stands for Tizen Compliance Tests, and it allows us to check the API for any regressions or if all the functionalities are provided by the board itself. Because doing something, something special, solution, it's always cost and you need debug and everything. So what was the point to develop the board? I wasn't possible to do it without it. The question was about the reason for the device itself. It's unification of the whole process. With those boards, we do not have to think whether it's minoboard or odroid or highkey. We've got a common interface to all of them. As you said, we don't want to have to reflash microSD cards and put it back to the device. From what I see, is that you are not exactly satisfied with the answer. Or maybe I just misunderstood your question. So about bootloader issue. We also test it. So if there is a new change in bootloader, we push those changes to target devices. That's why we don't want to break them if there is some crucial failure. Not all of the devices that we currently support do have some internal memory. Highkeys do, but odroids or minobords are not equipped with any internal memory. They can be booted either from microSD cards or in the case of minobords from the SATA port. But it's the basic storage device for all of the devices that we test on. Not all of the devices can be booted from other storage devices. Przestańmy odpoczywać niektóre biorę metalowe hardware borty z emulatorami. Kolejny, na przykład. Tak, więc... Czy to może być zrobione z tylko microSD karty emulatorami wytworzone w testowaniu? Trudno. Trudno. Kompletny hardware może być emulatowany. Dlaczego? My chcielibyśmy trzymać tak szybko, jak jest możliwe, do wytworzenia actualnych. To dlatego, że testujemy realne microSD karty. To dlatego, że użyjemy aktualne sbc w targach. To dlatego, że nie tylko testujemy kvmów, nie tylko emulatorów, ale też odwiedźmi borty z emulatorami. To tylko pytanie, ponieważ na przykład mamy wielki wytworzonych warunków wytworzenia hardware. Wytworzyliśmy, że nie wytworzyliśmy wszystkich w tym testowaniu i nie mogliśmy wytworzyć odwiedźmy, odwiedźmy i wytworzyliśmy. Nie mogliśmy mieć wystarczających wytworzeń hardware, dlatego że wytworzyliśmy do używania emulatorów w testowaniu. Dlaczego? Czerwony nasz setup jest zbierany na aktualnym hardware, a emulatorów nie zostały zbierane tak długo. Dlaczego? Dlaczego? Wytworzyliśmy, że nie było szkodnika wytworzenia hardware wytworzenia informacji i reżyserwacji. Wytworzyliśmy, że po jednej z tych aparty, wytworzyliśmy wytworzenie informacji czy wytworzyliśmy wytworzenie informacji, czy wytworzyliśmy wytworzyliśmy wytworzenie informacji. Po razie, i nie ma żadnych problemów. To dlatego, że w porządku, byśmy musieli położyć metod z położeniem log-fajów, prowadzonych przez basic utilities na Linux. Tak, pytanie jest o testowaniu inputu i outputu. Tak, tak, ten tct-framing sprawdzi, jeśli w porządku w porządku pracuje dokładnie. Ale tak, jak pamiętam nie ma benchmarku, jeśli to co oznacza dla transferu data i rozwoju kontekstu. Tak, nie testujemy tego i myślę, że powinniśmy na nasz list dokładnie. Tak, o publikacji testu testu jest dostępna na samej serwerie Tizenorg sdmx w porządku. Tak, ten tct-framing w porządku w porządku jest też dostępny na publikacji. Absolutnie. Tak, oczywiście. Wszystko w porządku nie jestem designer w porządku ale w jakiejś pytań możesz mnie dodać do mojego dnia czy po prostu do listu tizen na listu.tizen.org ten list jest najbardziej popularny. Te pytań jest najwyższa. Jeśli nie są inne pytania dziękuję za uwagę i do zobaczenia.