 Hello everyone, welcome to our session here at Embedded Linux Conference. My name is Tim Bird, I'm a Principal Software Engineer for Sony Electronics. I'm here with Harish Bansal, who's a technical engineer with TimeSys, and we are here to talk to you about embedded board farm APIs. So we want to talk about something we think will be really beneficial for automated testing and the improvement of the testing ecosystem. So next slide Harish. So I always like to put the abstract in, but we're going to skip over that. That's for people who look at the slides afterwards and maybe don't have access to the video. Our agenda today is I'm going to talk about kind of the problem statement. Why are we talking about board farm APIs? And then we'll give a little bit of an introduction to Embedded Board Farm Cloud, which is a development solution by TimeSys. And then we'll talk about some actual use cases for the REST API, give some details about the API and then wrap it up by talking about issues encountered and what's next. So getting started with the problem statement. So there are lots of tests that are available. But the basic problem is that there's no standardized way of running tests specifically on physical devices. So there are lots of different test frameworks. And there are lots of, there are board farm frameworks. There are certainly tools available to manage board farms and to manage the hardware that's in a farm. But there's no standardized way to access those, that software, that framework. So that different test frameworks and different tests can run in a lab independent way. The real big issue here is that every farm implements test infrastructure differently. So for instance, you know, Harish is going to present some information about the TimeSys board farm and the hardware that they've got in their lab, which is what we use to run our prototype on. I've got a board farm sitting here right next to me. It's off camera, but it's, if you've ever run a board farm, you know, that it's got a tangled mess of wires. I've got all kinds of custom hardware in there. I've got some off the shelf pieces for doing power control. And I've got a custom board for doing power measurement. And so the tests that are written in my lab that control that hardware are not going to run in someone else's lab. And vice versa. So QA departments for companies will write tests that will work in their lab with their particular infrastructure, but it doesn't work in someone else's lab. And so we can't share tests with each other. And since we're dealing with open source software, it seems like a real shame that we can't can't reuse each other's work. So our solution is we want to create a standard method for accessing board farm. And we think that what that will allow is for three big benefits. One of the first benefits is once you have that abstraction layer, you'll be able to allow the test infrastructure, the technologies to evolve separately from the interface. So for instance, I've got a certain type of power measurement board in my farm. But if other labs have different power measurement boards, they can evolve separately and introduce new features or or handle things in a different way. So the tests are not tied to a particular implementation or expression in hardware. The big benefit is that tests can be written to work in more than one lab. And that that allows us to create an ecosystem of tests rather than just have tests that are that are written by by one company or one individual. The other thing that it allows us for test frameworks that can work also with more than one lab. And we're actually going to demonstrate that working today. So the types of tests we're talking about are not just pure software tests. We're really talking about hardware software integration tests or end to end tests. So if you look at, for instance, if you were to do GPIO testing or serial serial port testing on a board, you could accomplish that by putting the serial port in loopback mode and testing both endpoints on the same device. But that's that's not really how the hardware is going to be tested or going to be used in the field. If you have a product that has a GPIO, you want to test it all the way through the hardware and out to an external device so that you can capture things like not just device driver errors, but things like mechanical failures or electrical failures that are part of that product. And that's very common to want to do that. So for all of these types of hardware kind of integrated hardware software tests, you actually end up having to control two endpoints. You have to control an end point on the device and then you have to control an end point that is off the device. And so I use the acronym DUT here, which is that I'll say that a lot, which is the device under test. So if you look at another example of maybe a hardware software integration test, you have audio playback, which or video playback. Again, you need to control two endpoints. You have an end point on the board that is actually generating the audio or managing it in some way. And then you have an end point off the board, which is a capture device to make sure that you're not getting any dropouts or time delays or that the quality is what you're expecting. And so again, you're controlling two endpoints. The same with power measurement. If you're doing power measurement off the board, you're going to control two endpoints. One is on the board where you're actually managing the workload profile that you're trying to test or the application. And then off the device, you're talking to some kind of board or measurement device that actually is taking power readings. And you want to analyze those after the test to see what happened in terms of maxes or total power used or something. And then another test that you can imagine is what we do at Sony is USB connect and disconnect testing to make sure that we have a robust error handling. And that's both kind of at the electrical level and for the stack, the driver level. Again, you need to control two endpoints, one on the board and one off the board. And so you will usually have hardware that disconnects and reconnects the bus. But those are not the only hardware software is not the only type of testing that can benefit from what I call multi device orchestration. So if you look at something like net perf, where you need to manage two endpoints. So you have a client on the board itself that is going to be generating network traffic. And you have a server that's going to be receiving traffic or measuring the performance of the traffic. You may want to, well, at a minimum, you need to discover the server address. And that's going to be specific to the lab. So you need to the test itself needs to know what the address of the net perf server is. But you also there also may be configurations where you want to make that net perf server specific to a particular set of nodes. So that you can reduce the amount of traffic collisions and make the test results more consistent. So you may need the lab management software to actually reserve the net perf server for a period of time and release it instead of just having it generally available that introduces inconsistencies in the test data. Another type of multi device orchestration that is common that we don't actually think of as multi device orchestration because we've been doing it so long is a boot test. In this case, you may need to manage several different devices. Well, you always have to manage several different devices. The classic boot test is, you know, you need to install a kernel on the root FS somewhere. And that that requires access to a storage controller. In the case of SD card based boards, you may want an SD mixer that you're communicating with to actually accomplish that. And then you have a serial controller that's actually capturing the serial line where you're getting the console output. And then you have a power controller to control the power to the board. So historically, we haven't thought of tests in that way as, you know, controlling multiple endpoints, but that's really what's going on under the hood. So our concept today that we want to introduce you is this REST API. And the REST API will will create an abstraction layer that any test framework can talk to and a test can talk to you to communicate with hardware and resources or or features that are inside a lab for a particular test and do it in a way that it doesn't matter which farm you're talking to. The same operations can be performed, the same data can be gathered. And so the test can be farm neutral, drilling down a little bit deeper into the diagram here. Next slide, Harish. Yeah, so we have if you look at what the API is actually controlling, you have some operations or some actions that are controlling the board in the lab. And but you also have operations that control those other endpoints that I was talking about. So you might have a power controller. You might have an audio video capture device or a GPIO endpoint. And those are all things that the REST API can manage for you in a lab independent fashion. And so the frameworks can access them in a lab independent fashion, but also the test itself that is actually running on the board can also communicate with or control those other resources. So that's the overall concept. And our REST API really consists of kind of three parts that I think are important. First, you know, you have the REST API itself, a command line interface and environment variables. So the REST API is actually a very kind of traditional web API. It's based on ATTPS with URIs that include actions and objects. And JSON is passed back and forth between the client and server. We actually did this as an extension to the Lava REST API. So it should be pretty familiar. And a lot of frameworks already have plumbing to deal with that Lava REST API already. It's a very simple API in terms of the amount of dependencies it has on external programs. So you can actually accomplish everything you need to with this API using just curl and JQ. JQ is a Linux command line tool for extracting data, individual elements from JSON data. So that's the actual API. That's kind of the most important thing. But we also provide a command line tool so that you have it has the same operations as the REST API. And it is suitable for automated use. If your test framework does things more with the command line and shell scripting, it's a little bit easier to use with that. But it's also much easier for humans to use instead of crafting these long URLs that have a very specific syntax. You can use the command line tool. It's a little bit more comfortable. And then the final aspect of the API is that there are some things that need to be communicated to the test program before it even has access to the API itself. So for one example, the address of the server where that is managing the lab. So some of those environment variables are some of those some of that data is passed as environment variables or it can be put in a file called etsy test config. So those are the elements of the API. And now I'll turn the time over to Harish to talk a little bit about the hardware that we used for the prototype implementation. Thank you, Tim. Hello, everyone. My name is Harish Bansal and I work for embedded board form team at times. So let me give you a brief introduction for of times is embedded board form cloud. This is the high level diagram of embedded board form cloud using the solution engineers who are sitting anywhere across the world, but they are connected to this cloud. They could access any remote port or embedded device and use it for their development and testing. These ports at the physical locations in lab, they are connected to a specialized hardware called zombies. These zombies with board, these could be distributed across different geographical locations, different countries, different buildings or different flows of of of a same building. And all these zombies with boards, they communicate with us centralized server, which we call embedded board form master ebf master. And engineers connect to this ebf master to access the remote board and embedded board form cloud features. This embedded board form cloud master, it's a Linux machine which controls all the zombies, which are on the cloud and the devices connected to it. It runs a web application inside a Docker container, which provides centralized board management, user management, zombie management, image and file transfer management. It has also integrated a lava test framework. So on any of the device which is connected to any of the zombie, you could use it for running automated test. On the right is the picture of actual zombie hardware. It's divided into two parts. One on the left is a zombie section for embedded board form feature. And on the right is an app or a test server part, which is used for testing and streaming. The box on on the left, the part under red box, this is a zombie part. Here devices, serial ports are connected at the top and it has internet ports for bringing the devices onto the network. On the right, one under blue box is the app or test server section. It runs Linux and this would act as an additional lab machine for running tests like IPERF or netPERF, which require additional endpoint in the lab infrastructure. There are ports where you could connect peripherals like JTAC for debugging your hardware or connect a webcam to see live images of your board, embedded board or DUT. On the bottom left is another lab controller. This is called input output controller IOCX. It's an optional hardware which sits between a zombie and device. So each zombie could connect up to four devices and with each device, one IOCX is connected. On top of IOCX, there is a mounted power relay switch. Through this, you could control, one could control board's power. Then it has USB and Ethernet hot plugs. These hot plugs are like switches. So many a time, one needs to on-demand connect or disconnect peripherals from the DUTs or boards. Those actual peripherals are connected on these hot plugs and then using a web UI of embedded board form or REST API, one could on-demand connect those to the device site or remove it. For boards which puts out of SD card or use SD card as one of the board media, there is STMUX built-in. So this notch with the ribbon cable, it goes into the SD card slot of the embedded device or board and the actual SD card is connected at the back of this IOCX one which is highlighted with the red box. And then remotely, you could switch the SD card to the device site or switch it to the zombie site. Once it switched to the zombie site, you could manage the SD card content, format the SD card, create different partitions, upload or download files from the SD card or flash a new device image to the SD card. Then it has I2C and GPIO connector, which I'm going to talk about in later slides. This is a device dashboard of the web user interface. Here you could see all the devices or boards which are connected across the embedded board form, which board is connected to which zombie board or device status who is currently using it. And if any device is free, you could assign the device to yourself and start using it. Once you assign the device to yourself, you get access to the device console. If you get access to device serial console, if the device supports is running SSH server, you could access SSH console and if it's an Android device, you could even access its ADB console. If device is in Hangstate or it's powered off, then we could use this power commands, power control, off or on reboot. The commands that you see, it depends on which power, remote control power switch are you using? The one at the top, it's not necessarily that you should use the one which is at the top of IOCX. With time system board form, you could connect any remote control power switch, whether it's IP control, SNMP control or relay control. And the commands which that power switch supports those show up here. So the commands could be off with delay, on with delay, A plus, B plus, whatever that power supply, power remote power switch supports. You could turn on streaming. You could see video streaming or audio and audio streaming, one could do SD card or network boot image management. On the device page at the top, there is a menu, we call it IOCX menu. So here, SDMUX device green shows the SD card, which is actually physically connected at the back of IOCX. It's on the device side. SD card, if it's red, that means the SD card is connected on the zombie side and it's ready for you to manage the SD card content. On the right of this SDMUX are hot plug buttons, red means off, means disconnected, greens means they are connected back. So Tim is going to discuss in detail GPIO test. So let me describe GPIO setup in the board form. So at the left bottom is IOCX. This IOCX has eight GPIO lines, which could be connected to the DUT or device. And so out of these eight, the six pairs of line ground comes out of this GPIO connector. And two lines comes out of this I2C connector. And all these GPIO lines could be controlled using REST APIs. So let me go ahead and now describe a prototype use case. Keeping in mind the GPIO hardware capabilities in the embedded board farm cloud. We designed a test to test GPIO on the board. And we're using the REST API. And what we're going to show in our prototype use case is a couple of different things. We're using from both the time sys web UI, you can access features on the board and perform the test. And you can actually run it as a lava job. And you can use Fuego to also initiate that same job and perform the test. One of the things that we didn't quite finish, but that is pretty close is also running that test in the Sony embedded board farm. We're pretty confident that it would work, but time sys did a better job of finishing up their implementation than I did. So but looking at how the test is actually going to run. So we're going to show you a couple of different things in a video coming up. So you can run the run these tests or use the API in several different ways. So you can do manual execution. So if you want to get onto the board and actually issue the GPIO commands yourself, or access the lab resource yourself, you can do that manually through the command line tool, or through the console that's available. You can run a test script, which basically does those same types of commands, except using the REST API directly from the script. And that's you'll see. And you can run that test manually, or the next the next phase is to put it under test automation. So we will show you a test running from a lava job, and then from Fuego. And and that and that puts the test encapsulated in kind of the larger test framework that does things like schedules the job and presents the results at the end. So with that, let's let's go ahead and look at what we did. So here, in this video, we are showing three different ways of running our GPIO test script. This is first example of running the test script using device web console. So we logged in into work from cloud web interface on the device dashboard we are searching for a device. We allocated the device. Now we're going to the console page. So device console page here we are straight into the directory which had the test executed the test and we could see test ran and it printed out the test result. This is second example of running the same test through a command line tool. So ebf is the name of the tool. My devices command list give list of all the devices which are currently assigned to this user. This with this next command, we are going to run execute to run execute some command on the board. So this ebf tool, you could install it on your machine and once installed, you could access any remote board. So this video was prepared by my fellow engineer Ankit Gupta. So he installed this ebf tool on the system. So from his system, which is physically in Delhi, he's running command on this Raspberry Pi board, which is connected to a zombie in Pittsburgh in US, in our US office. So if any first ran ls minus al command to see whether the test script exists. Now he's running the actual test script using SSH run command on this Raspberry Pi 4 GPIO board. It's executing on the server on on the board and it fetched the test script output. And we could see directly on our on on the local machine. This is a third example. This should be interesting. So so far, we showed you manual ways. So now we are going to run the same GPIO test through test framework Lava. For this week, you created a Lava test drop PQ curated, we are going to submit that manual test job YAML. So here we copied it. This is the GPIO test we are going to run. We want to run it on the Raspberry Pi 4 device type Summit Lava job. And it assigned Raspberry 4 GPIO device for the test job. It's running. You could see test job blocks coming on the screen. Close to it's done. We could see same test script output in the desktop lock. If we go to Lava result, we see GPIO test passed. Okay. So now for this next section, this was from the Jenkins interface. So Fuego uses Jenkins as its user interface and we can see that I've defined some tests that are part of the Fuego infrastructure. I'm going to do a Jenkins build, which is the same thing as a test execution. And so I'm using the same the same REST API to perform some operations with the board. Fuego does the test in a series of phases. The pretest phase is where we check connectivity and do some some basic board sanity checks. Then we're actually going to deploy the test to the board and execute it and we'll get some results back. I think it's important to note that I do not have SSH access between my lab and the zombie. I'm using the REST API. So this is there's SSH between the zombie and the board. But these are all operations that are going purely through the REST API. So there's no direct connection between between the labs. And you don't have to open up holes in your network in order to let stuff through. Anyway, this is showing that the test passed. Everything worked great. And so that's it. It was a short demo, but we're pretty excited. There's been a lot of work over the last couple of months to get that all working. And so we think that this heralds the dawn of a new age of remote testing for automated for embedded Linux. So now I'll let Harish, I think he's going to talk a little bit more about the API details. Thank you. So let's get under the hood and see how it's working. Here is the actual GPIO REST API. So it consists of GPIO command, GPIO pin pattern and GPIO pin data in the URI itself. Current aid supports eight different GPIO commands, four for bit mode, read, try, get mode and set mode, and four commands for mask, read mask, write mask, get mode mask and set mode mask. GPIO pin pattern, it's lap dependent. So in our case, this IOCX controller has eight GPIO lines. So lap pin numbers varies from one to eight. And pattern mask varies from one to 255 in decimal. GPIO pin data, again, it's lap dependent. This API, it returns output in JSON, a JSON object result with value success and data value would depend on what command you run, GPIO command you run. In case the command fails on the server, it returns result as fail and why it failed, that failure reason comes under message. This is a section of actual GPIO test script. Here it GPIO device, GPIO pin 20 was connected to lap controllers pin number six. It sets the pin high on the device. And then using the rest API, it fetches the value of lap controller GPIO pin. Then it does a match between the two. If the value matches, test case passed, it equals okay for the test description. In case there's a no match, test case fails, and it equals not okay with test description. embedded board from command line tool, EPF tool, which we showed in the demo, it's implemented using the rest API. It could be used for automation or interactive manual use. And all the EPF CLI commands they are purely using the rest API. So in this, I have listed some of the commands and you could see how these commands are directly mapped with the rest APIs. Okay, so now I think I'd like to go to a wrap up and tell you a little bit about some of the issues we encountered and some of the next steps. So a couple of things that I think are we encountered that were interesting in terms of creating this API. I think it's important to note that kind of different parts of the test framework and the test use different parts of the API. And so that's that's useful information because it helps you understand what parts of the API kind of are needed by by different sections of the overall CI loop. So for instance, the test framework is going to do use actions like run upload and download. But in our case, the GPIO test itself used things that were dedicated that were kind of targeted at a particular resource in the lab hardware resource, things to do set direction read and write different frameworks put control of operations in different places. So it's important to kind of separate them out and understand which parts of the API are being used by which parts of the different test frameworks. One of the things we did determined that is that we need to extend the API a little bit. There's predefined data that is lab independent that you can kind of put on put into the test without having to worry about it. But there's also discovered data. So in our example, we have a GPIO endpoint on the board and a GPIO endpoint in the lab. And you really need to discover the mapping between those two. So we don't actually in the API right there. The demo we had today, we hard coded those, but those are going to be different per lab because of it depends on the wiring that you've done as you've set up the hardware in your lab. So we're learning the types of things that we're going to have to do to extend the API to cover use cases and make the test truly lab independent. The other thing that we discovered is just impedance mismatch between the expectations of different test frameworks. So as an example, the way that the test operates in the in the time sys lab, they only needed a single file to be copied over Fuego. So a lot of Fuego tests use a recursive file copy. Currently, the rest API only supports that single file. And we're able to work around that. But we want to look and see what are the actual requirements from the different test frameworks, and how they operate and continue to refine the API. And then the last thing is you need to integrate all of this with your larger CI loop. So there are actions that you perform on your boards, provisioning the board, and getting stuff set up that you also need to take into account. So what's next? So we've demonstrated the basic concept and we're pretty excited about it. But we need to we need to go to the next step create API is rather lab resource types. So we're pretty sure that it's, that it's not an endless task ahead of us. There's a lot of resources. We did a simple end to end task where we're just controlling two endpoints with basically ones or zeros and some directions. For other types of lab equipment, you're going to need different APIs. But we think that a lot of the types of things you're going to be doing with the lab equipment is going to be generalizable, you'll be able to generalize it to just a couple of simple commands. So for instance, for power management, audio capture and video capture, you're going to need a verb for start capture and capture and then get that log. Now the actual so obviously audio capture and power measurement return different types of logs. But we don't think that the API has to actually deal with the specifics of the lap of the kind of a log type. But we wait, we may decide differently. So we, we may decide to include some resource specific type actions to support. So for instance, in power management, instead of just returning get log and having the framework or the test analyze the log, we may introduce a verb like get max power, where the driver for your power measurement device can analyze the log for you and return a single number that that has meaning for the test. But overall, we need to run a bunch of different tests and see what issues crop up. The important thing and the reason we're here today is we want to convince other labs and frameworks to adopt this API. That's how we're going to get to a place where we can have an ecosystem of tests that we're all sharing with each other. And so we'd like to profit from that ecosystem, start sharing tests the same way that we're sharing open source code, we'd like to be able to share open source testing artifacts, the tests themselves, the results, and all that. And we think that this is a way to start with all of this type of stuff. You have a chicken and egg problem, someone's got to go first. No, you that's fine, or you can go to the next slide. So we know that people are not going to write tests until the labs have this in their farms, and people are not going to want to put this in their farms until people have written tests. But someone's got to take a first step. And we feel like we've got a good start on this. There, there are schema definitions and other materials related to this project that are going into this repository here then GitHub. And we are very excited to have people give us feedback. If you want to join the lab, join join this effort, please communicate with any of us. And we would love to get additional labs, especially of different types of hardware involved and try to do some testing to make sure that we're covering everybody's use cases. And with that, I will thank you for your time and we'll move on to some comments. So any any questions or comments?