 So, hello, good morning everyone, welcome, my name is Krzysztof Pasiak, I work for Samsung R&D Institute Poland and today I would like to share with you my story about grabbing audio and video on a board far. So, generally I know this is very early, so if I fall asleep during my talk, please remember to wake me up in the end because I have to pass the microphone to the next speaker. And I hope that if you fall asleep, it will be really a comfortable sleep. Okay, so now let's go to our presentation. So the agenda for today. First of all, I would like to tell you a few words why people run a board farm. Generally it takes a lot of time and a lot of money to take it, not only to build it, but also to maintain it. Then I will go through some typical setups. In particular, I will tell about Lava and I will tell about Slavstock, which we are building in Samsung. And then I will go to the HDMI Pi, which is my proof of concept of how to grab audio and video on a board farm, how to use your target remotely with full audio and video support. Then I will go to Futurework because this is only a proof of concept. It works. I would say it works perfectly, but has some issues. So I think that this can be done much better. That's why I suggest some future steps. Then I will go to the summary and the Q&A session in the end. So generally, what is a board farm? Building a board farm means that you are taking some kind of room and put there a lot of boards. Why? So the first use case is to run automated testing. We need testing during whole software lifecycle, not only in the end when we ship our product to the client. In case of projects like OpenStack or OnApp or big projects related to the cloud or server infrastructure, there is very high pressure on the CI system. In terms of kernel, we are doing a little bit too less of testing. So that's why we should put more pressure on it. And manual testing of kernel is really time consuming. It's not easy to test your patches every time when you are sending them. It's not easy to take the kernel every minute because there is a lot of comets coming to the Linux tree. So we should really test it if this doesn't break our hardware or maybe not our hardware, but putting a kernel on our hardware. So what we can do? Well, we can try to use emulator, but it's not always the real hardware. There are some bugs that are really hardware dependent. So that's why we sometimes should use the real hardware and try to test our changes and newest kernel versions on that hardware. Another use case is sharing the hardware. Generally, there is always not enough hardware, even if you are working in a hardware company, developer boards are always, always busy. So that's why we are sharing boards among many developers. If you are working in the same office, it's really easy to do this. You can just pass the board to your colleague. But still, you colleague has to recreate your setup. Setup, serial, set up all the stuff that is needed to play with the board, all the stuff that he's working on, etc. What if we could say that this board is always ready to be used and you can just allocate a time slot and everything is connected? Another use case is when your target is secret. Well, if you are working with consumer electronics, it's not always the software that is very secret, but the look and feel of the end device. So you may try to bypass this by providing developers with so-called black box electronics. But what if you could skip that step of producing these black box electronics and just produce the real targets and put them in a closed room when no one have access to? And developers would just get a remote access and try to deploy their software and run it, play with the board, as if it would be on their desks. Next item is having access to the remote machine. Well, many companies scatter all around the world. Or even if you are only based in, for example, UK, you may have a client from China, you may have a client from Japan, etc. If you have only one target, then probably it's going to be shipped to your office and you will work on it. But what if you work with multiple products? Come on, it's not always easy to get an access to the exact board that we had to fix the issue on. Because, for example, Samsung sells mobile phones with different chipsets in different corners of the world. So if you are based in USA, it's not very easy for you to get, for example, Galaxy S9 with the chipset that is being sold in Korea. You have to order it in Korea, wait, for example, one week to ship it, etc. It not only costs, but it also increased the time between finding a bug and fixing it. So it's not easy. There is a lot of use cases, but it's not very easy to have a board farm. There is a lot of challenges. The first of them is that every board is unique. So if you are working with multiple boards, but you'd like to have a unified access, you need to somehow figure out how to unify it. It's not easy. There is some supporting hardware, I would say, but it's either expensive or either not very available. And there is this maintenance cost. If you are having a complicated setup, if you are using multiple targets on the same machine, if something gets wrong with one board, you may end up with a situation that you have to disconnect all other boards from that machine and try to debug the issue. This may block a lot of work that is being done by other developers. Another issue is stability. As people say, USB works. Yes, it works for one day, two days. But after a week of constant work on a board farm, it starts getting really messy. You will get a lot of errors. Your targets will stop enumerating. So USB is not as stable as the internet. Then there is a scalability issue. You always start small. You start, for example, with a single board. But really fast, you are trying to connect additional boards. And if you are doing this in the wrong way, then you quickly end up in a situation that you are the only one who knows how this stuff is configured. If someone comes to the room, he says, oh my gosh. What this cable is for? I have no idea. What happened? It's just not working. And there you've got the phone in the middle of the night that, oh, come on. I don't have to come to the office because we don't know what happened. And serial for board number 32 is not working. So it's not easy. It's just like running a server room. So what are the typical setups? How people deal with it? The first one and the oldest one is lava. Lava stands for linear automated validation architecture. Basically, lava allows you to run a board farm with a setup that there are a number with a distributed setup. So the main server is lava scheduler or lava master. This is a server where you just submit a job. The job is described by Yamu. And in the job, typically, you just download the image, flash it to the device under the test, boot your board, run some kind of test scripts, and collect the artifacts to present this to the upper layer. So this is all described in that Yamu job language. And this lava scheduler selects the dispatcher that should execute the test on some particular board and pass the description to this dispatcher. Dispatcher is a machine that has some device under tests connected. Usually, you will get a PC with some kind of power relays with some kind of serial adapters with USB hub or maybe internet switch. It depends what your boards offer. But generally, you have one lava dispatcher with multiple boards. And obviously, you may have a number of that kind of laboratory. So for example, one may be placed in USA and the other one in Korea. So there is also a concept of lab in a box. And basically, it's a concept from Bay Libre from last EOC. What they did is they containerized lava dispatcher and lava master and put them on a single machine. Obviously, this can be scaled up. And you can just deploy master on the other machine. But the concept is to put everything into one single PC box. So what we've got here is the power delivery unit, some kind of motherboard. We've got ACME probe, which are used to measure the power consumption. We've got some number of device under tests. And everything is closed into a single box. It's nice. It looks really nice. But the problem is that still, if something happens with a single board, you may end up in a situation that you have to disconnect multiple boards to debug the issue. Because for example, if your USB stops working, it's not that only one target which is connected via USB will be unavailable. All which are connected to that controller are going to be unavailable. So still, you may need to reboot the machine, which means that you have down all the boards. So that's why we took a different approach. We decided that a single device under test should be connected to a single PC-like machine. Well, I say PC-like. I'm saying not PC-like, you know, with the huge box and huge amount of memory. No, I'm just telling about a small single board computer called Nanopi. So what we did is we developed an add-on to the Nanopi Neo, which simply unifies the access to almost any board. And this is the very bottom layer of our solution. So we may have similar devices. They are called MaxPies. And you may have different device under tests. And the benefit of this approach is that to the upper layer, you provide only internet connection. And here you connect the power supply. So the upper layer doesn't even know what kind of target is here connected or how to flash it, how to play with it, because it's all done with the Moogspie. It provides the unified kind of API that allows you to access the device and execute some kind of tasks. For example, do the power cycle. For example, flash the image. Or for example, deploy something on the target. But this is only the very basic building block. The next step which you have to take is how to allocate this device to the developer or to the test system. So that's why we developed Boruta. And this is a kind of equivalent of lab grid that someone of you may know. What it does, it allows to allocate the target for some certain period of time. So for example, you can have a request that you would like to get access to the Raspberry Pi. You may also pass some capabilities of that device. For example, it should have camera connected. It should be able to capture the video or something. And Boruta provides you an SSH tunnel to some particular device. But giving an access to the device, it's not always enough. It's fine if you'd like to play with the device, debug something for a short period of time or something like this. But if you'd like to run your automated tests, you need another system. That's why we developed Veles. And Veles from the upper side is almost compatible with lava. So it also takes the YAML job description. And then it uses Boruta to get the access to the hardware and deploy and run tests. All the stack and all the design for that board is open source. So you can get it from our GitHub. Because this talk is really focused on the hardware, because I will be talking how to grab audio and video in that kind of setup. Let's focus on Moogspi now. So what Moogspi really is. So like I told earlier, this is a single board computer called Nanopi. Why we choose Nanopi? Because it was cheap. It was cheap and powerful enough. So there was no other reason to choose it. If you'd like to use different single board computer like Raspberry Pi or anything else, OK. Just develop an adapter and connect it. There is not a problem. We are not stick to the Nanopi or we are not depending on any of its features. So we've got the Nanopi Neo over here. And this Nanopi Neo is connected to the microcontroller, which provides all the low level interfaces of the board. So what we've got? First of all, we've got power control. So you can control if the device is powered on or off. You can not only control this, but you can also measure the power consumption. So it also contains the ACME props. The next thing is the SD-Max. What SD-Max is? Most of single board computers may boot from the micro SD card. It's quite convenient to flash the micro SD card via USB, for example, or to execute Pixie Boot or anything like this if your bootloader is working. But if you are flashing whole images, including the bootloader, you have no guarantee that the bootloader is not broken. So that's why we decided that we need an automated way to recover our device in case when we flash a broken bootloader. That's why there is micro SD card demultiplexer. Demultiplexer means that the micro SD card, which is inside the MOOC spy, can be connected either to the device under the test or to internal micro SD card reader. So if you switch this, you may always flash the whole image of the micro SD card and recover from almost any state. The next feature is diaper. When your board is just booting after connecting the power, it's fine. But there are some really nice designs that some boards require, for example, pressing a button to boot after connecting the power. So to automate that kind of stuff, MOOC spy contains diapers, which stands for dynamic jumpers. So you can emulate the button press things. There is obviously the built-in serial support. And there is also the USB serial switch. Because if you are developing something for the mobile phone, you may end up in a situation that serial is on the same pins as USB. So you have to use the suitable resistor to choose whether you'd like to connect your mobile phone as a USB device, or you'd like to get the serial access to the kernel or to the bootloader. There is also display, because it's really nice and easy to deploy a device. But when it comes to the maintenance, you need to have some kind of way to identify that, for example, this is the broken board. If you have like 100 of those boards, it's really nice to have at least a diode to signal that this is the board which is broken, something is wrong, this is the board you should check, instead of just going through all the boards and looking which one is the broken one. It's also extendable, which means that we provide an extension board to which you can connect your own add-on. When we've been developing this, we really think that people will do crazy stuff with it. We don't know all the use cases. For example, we do not support TV over here, but there is an add-on developed which provides you the remote control for the TV. We don't have any speaker or microphone here, but there is an add-on that provides you the speaker and the microphone so you can, for example, test some NLP stuff. And in the end, there is no video or audio input on that board, but you can develop an add-on that provides you this. So what is on the board in terms of audio and video, there is ADID injector. So what does it is? Well, this is how the board looks like, and you see that there is an HDMI connector over here. But this is only a fake HDMI. What does it mean to connect a monitor? Every time when you connect a monitor over HDMI, it sends the information about the available resolutions to the board. So the board waits for that information, and when only it receives it, it starts playing the video. There may be nothing on the other side, but the board will think that the monitor is connected. Why we need it? To use cases. First of them is that not every board is really happy to boot up without monitor connected. And the second use case is to just simply make a screenshot. Before we had this add-on for grabbing audio and video, we were doing just software screenshots by logging to the board and making a screenshot. But to make a screenshot, you need to somehow emulate that the monitor is connected. Otherwise, you will get just blank screen. So that's why we have developed this ADID injector, and we put this on the MoogSpy board to make this happen. So this is the MoogSpy. But now let's go to the meat. Let's go to the grabbing audio and video because as you saw, there is only some kind of fake stuff to emulate that. So what can we do? First of all, we could just go to the shop and buy an HDMI grabber. They are really available on the market. It is not a problem to buy them. The only problem is that it costs around $300. And still, if you buy a single device, it's not enough for your board fire. Usually, you need at least 10 of them or maybe 100 of them. It depends on your board farm size. And still, even if you use that device, you need to somehow stream the video over the network, which means that potentially you may need to convert it and converting a video on a single board computer like Nanopie, it's not really something you'd like to do because it takes a lot of time, so it would increase the delay. If you'd like to just make a screenshot, if you'd like to just use this to, for example, make some videos of the board, something like that, so only the automated test stuff, okay, it will work. But we should consider not only the automated tests, but also remote access for the developer. The other issue is that many of those grabbers requires USB 3.0. And obviously, a single board computer for less than $10 is not equipped with that. So it may be not easy to connect it. Another issue is that those grabbers comes in different prices, in different forms. Let's say, and some of them use CPU very extensively. And this, once again, on a single board computer may become an issue. So that's why I picked up an idea from Fosnem, presented by one of the Intel guys. He mentioned that there is a really nice device. It's called LKFOW3738. What is this? Well, this is HDMI extender. So what is it designed for? If you enter the shop, you may see that most of the TVs are playing the same video. So if you'd like to connect every of those TVs to a separate media device, you would need a lot of HDMI cables and those are length limited. So to overcome that length, an extender like this one has been developed. The pair of devices consist of the receiver and sender. Receiver is a device that takes HDMI output, yeah. It has HDMI input and provides the MPEX stream over the network to any other device. And then to the TV, you may connect the other side, which takes the MPEX stream from the network and provides HDMI output. So it's really nice solution. And the funny thing is that you can easily grab this video stream, not only on that particular device, but for example, on your computer. And another really nice thing about it is that it costs $30. So it's extremely cheap. So what is the wiring of our setup? So we've got the Mux Pi. And this is our, let's say, remote computer for now I use the Ethernet to USB and I'm connected via Ethernet to the Nanopi Neo. This Nanopi has this Mux Pi connected and the power supply five volts. The power supply here is delivered to the device under the test. This is the place where I can decide whether this power supply is on or off. There are also two jumpers on the board. First of them is used to decide whether HDMI extender should be power on or off. The second one is used to control quality of the image. Originally, this board comes with a button that when you press it, it just changed the resolution. There are two resolutions available. It's not really stable. It doesn't always work, but it is possible to limit the resolution from full HD to the smaller one, to the 720P. Then we've got our device under the test, which is connected via HDMI to this HDMI extender and HDMI extender is connected via Ethernet to the Mux Pi board. This Ethernet is available on the Nanopi so Nanopi can take the input from here. I mean the MPEG stream and stream it to the remote machine. Device under the test has also a USB connected. So Nanopi Neo here, in this case, is used to emulate both keyboard and mouse to provide also user input from here to the device under the test. You will see this in a demo. So this is how it looks like when all everything is connected. You see the HDMI cable, Ethernet cable to the Mux Pi, power supply, Ethernet to my laptop or to some remote machine. There is also micro SD card, the multiplexer like I mentioned earlier, and there is the USB cable, which goes to the Nanopi Neo. So how do we start? First of all, we have to somehow capture this MPEG stream. It turns out that by default, this device is shipped with IP like this one. So we have to set up a suitable IP on your computer and then just try to grab the multicast group identified with that IP. It can be captured by almost any software that is doing video processing, for example, VLC, for example, FFMPEG and many others. Generally, I got the best results by using VLC. Maybe it is an issue with FFMPEG parameters or FFMPEG stream. I don't know, I'm not FFMPEG specialist. That's why I just went with a simple solution. I used VLC. And it turns out that VLC parameters really matters. Depending on those parameters, you may control how big is the delay and you may control how often your image will get some glitches. Unfortunately, there is a trade-off. Less delay you would like to get, more glitches you are going to get. Sorry, nothing is best. VLC version also really matters. By default, Ubuntu 16.04 that I used to test it was shipped with VLC 2.something. Unfortunately, it didn't work in my use case. That's why I had to recompile the 3.something. And it works. And the Nanopie kernel version really matters. I mean, if you use two old kernel, then you will end up in very big delay. And if you use two newer kernel, then you will end up with a lot of glitches and there are a lot of broken UDP frames which unfortunately destroys your video. Okay, so solution. If you'd like to just take a screenshot, so you would like to have a hardware screenshot to check if your UI is booted correctly, then it's really enough to use FFMPEG on the Nanopie and you're done. But if you'd like to consider also use case with a remote access, you need to somehow stream the video. The simplest solution is to just use SoCut and receive the video stream on a OneCard and just stream it to another IP. So the issue here is that it calls a really high CPU load on the Nanopie. And the second issue is that it really increased the delay. So we have to think about something better. And the something better is to do the streaming and the routing inside the kernel. So that's why you may use SMC route and the simple rule of IP tables to just change the sender IP and stream this to then a remote machine. Unfortunately, this is also not a perfect solution because I was unable to change the destination. If we've got here some kind of network guy who knows how to do this because I tried to use IP tables for that, unfortunately it was not working. So if someone knows how to do this, patches are welcome. There will be link on the other slide. So, summing up, audio and video, what do we need to make this working? First of all, we need VLC version 3.something. And the good parameters I found was network caching 200 and clock jitter zero. It provides a reasonable delay with not so many glitches, so it worked fine at least for me. I use the kernel of 4.11. It worked and the delay is not so big. It turns out that the delay of the network card which is connected via USB is smaller than with the built-in one. I don't know why, but this is what I found out. And I use SMC route 2.0. Okay, but having an audio and video in an MPEG stream on a remote machine is not really enough because you may need to play with the application that is running on the device under test. That's why you would like to provide input events from your remote machine to the device under the test. To do this, we use Nanopinio. It is equipped with USB device controller, so effectively it may act as any random USB device. Fortunately, Linux kernel provides the implementation of human interface protocol, which allows you to really easily simulate both keyboard, mouse, joystick, anything like. To set up everything on the Nanopi, you have to execute that kind of script or just use libusbgx to set up it. So keyboard. How do we emulate keyboard? After executing that script from the previous slide, you will get two new device nodes. They will be called dev hitg0 and dev hitg1. The hitg0, in this case, is responsible for emulating the keyboard. So how do we emulate the keyboard? We emulate a keyboard by writing a report descriptors which are defined in human interface protocol to that device node. So what is the format of the report descriptor for a keyboard? It's relatively simple. The first byte, please note that the order here is reversed. So this is the last significant, the least significant byte bit. And the first four bits are control, shift, alt and GUI on the left side of the keyboard. The next four are control, shift, alt, GUI on the right side of the keyboard. And then you may send from zero, there is a zero placeholder. And then you may send from zero up to six keys. So in this case, this particular report descriptors means that we have a shift pressed on our machine and a button which looks in that way. So effectively a character which will be recognized by the device under the test would be a quotation mark. So the next thing is mouse. It's not really easy and convenient to emulate a mouse which is being passed to the remote machine. Why? Because every time when you move your mouse, you should send a new report event. So you will end up by really filling your network traffic with a lot of unnecessary movement events. So that's why it's much more convenient to emulate a touch screen. This way your report events are sent only when the mouse button is pressed. So for example, you don't have problems that mouse is moved outside of the application zone because obviously you don't want to send input events when the mouse is outside of the video zone. So that's why, so I emulate the touch screen and it's relatively easy. It also have a report descriptor. The first byte is the number of fingers. So this is the array. There may be multiple finger descriptions. Finger ID, some random integer because I use only one finger. Estate, whether it is pressed or not. And the X and Y position of the event. So effectively when you click a mouse on somewhere on the video, you have to do recalculation because they have their own maximum and minimum. You have to do recalculation from the video position to the human interface position and then send the suitable report descriptors. Obviously all those commands should be executed on a Nanopie. So this device node is visible on a Nanopie Neo. So that's why we have to use SSH to send this event. So how to put this all together? Well, I wrote a simple application with QT library. Why QT? Because it was simple, nothing personal. I use QSSH to execute all the commands. So to set up the heat gadget, to set up the video forwarding and to emulate the human interface. I use QVLC to display. This is basically just the QT widget which allows you to embed the VLC video player inside your application. Nothing special. And all the thing is open source together with these wiring schematics you saw. So you may just pick it up and try to play it. Obviously patches are welcome. So live demo. Okay, let's see if it works. So what I've got here is the same setup as you saw in a picture. This is the Moog spy on the left side. The device under test which is Raspberry Pi in this case and the HDMI grabber which is above. So by default, the grabber is shipped with the really nice metal black box but it has some one issue. The cooling is not really good. So that's why I had to remove the case and put this in a room with really good air conditioning because after like two hours or something it's getting really hot and this generates a lot of video issues. Okay, so the application is not very fancy. As you see, there is just one window and there is just one menu. You just connect by providing the IP address of the Nanopi and the port to which you'd like to connect username and password which is visible for everyone. And you just click okay. So when you click okay and you have successfully connected to the Moog spy, you have to click okay once again. Surprising. I have no idea if you heard it but there was some noise from the Moog spy. It means that something gets connected. We enabled the HDMI grabber and we enabled the power supply to the board. So what is now actually happening is Raspberry Pi rebooting. So in couple of seconds, we should see the screen from the Raspberry Pi here. As you see, I'm using two small power supply. So this is already an output from the Raspberry Pi and now what we can do is we can for example, click and you see that there are two mouses or two mouse pointers. One of them comes from the Raspberry Pi, the smaller one and the bigger one is my desktop mouse. So every time I click the Raspberry Pi mouse moves. So for example, I may now try to close that window. Ta-da, it's closed. I may also open for example, the home or something and maybe try to play some video. It's playing with the overlay. So if you'd like to take a screenshot, you would probably not see this video because it's a hardware overlay. And one of issues of that application is that sound is not always working like it is now not working. But you need to restart the grabber and restart the board and you could get this in sync. As you see, the video is playing quite nicely. Okay, so issues. First of all, lack of mouse tracking. This may be a kind of issue if you'd like to for example, lay a game on your device under tests like Minecraft or something. I didn't manage to make internal multicast to unicast conversion. So still Moog Spy generates a multicast signal. This is kind of an issue if you would like to use more than one grabber. So this is our use case. So really if someone has an idea how to do internal multicast to unicast conversion really purchase or welcome or maybe just try to find me in the corridors. There is some kind of delay. Probably didn't notice it but there is like one second or maybe half of a second delay between my click and when mouse pointer moves. It's kind of related to the grabber itself because it does a lot of buffering. When you are playing just a video over the network like in the shop, the delay doesn't really matter. What is important that all TVs play the same content in the same time. So that's why they are doing here some kind of internal buffering and they need to flush that buffer before I get a new video. There is also the video quality. Well, the issue here is that it's not really possible to tweak it. You can get it, it's reasonable I think. It's more than enough to work with it but you cannot tweak the parameters if you'd like to use for example less network bandwidth or you have one gigabit internet and you'd like to stream the best possible quality. It's just fixed in the hardware. You cannot really edit it. And another issue is stability. The stream itself is not a pure MPEG stream. It contains some proprietary changes and the VLC or the lower layers are not always able to get for example audio and video in sync and this end up in a situation that you for example lose your audio. So we have to do something with it and that's why I suggest some future work. So if someone is interested in this idea you can just take a proof of concept and start playing with it. But what I found more interesting is that you can use a kind of open source open source and open hardware project from Australia to do something better. Like I mentioned, this grubber is really cheap but it has a lot of stability issues and it's not really open hardware. So that's why you can use the team videos and the team videos provides a board which is called Numato Opsis and basically this project is used to record the conferences like this one. So it has multiple HDMI inputs and multiple HDMI outputs and it may provide also the network stream and the USB UBC stream. So if we try to collaborate with them and reduce the hardware because the issue here is that we cannot take this hardware as this because it costs a lot. So we could reduce it, use less powerful FPGA, use less hardware in terms of HDMI files and just take it and use it and create the open hardware version of that grubber. The really nice thing is that they are not using the VHDL or any FPGA-related language. They use just Mijan, which is a Python toolbox for creating FPGA bitstream. It's really nice. If you didn't play with it, I really recommend to try it. So to do this, you have to get access to the Numato board and limit the bitstream to the FPGA to just the things we need and then find cheaper FPGA and redesign the board to handle it. So summing up, creating a board farm is challenging. There are two approaches. Basically, multiple-dude single controller or single-dude single controller, which is the later one because we think it's better in terms of maintenance and there is HDMI, this HDMI extender may be useful for testing but remember it may have some issues so you may try to fix them or try to go with the open hardware video grubber from TeamVideos. So thank you very much. I'm not sure if we have time for questions. Nope, okay. So just find me in the corridor. If you are interested in Slav, there are some brochures you may try, you may take them and read about it. Thank you. Thank you.