 Hello, everyone. Welcome to embedded Linux conference and welcome to this session. I'm here with Harish Bansal. My name is Tim Bird. I'm a principal software engineer at Sony electronics and Harish is a technical engineer with time sys. And this session is about board farm API is for automated testing of embedded Linux specifically this is an update to a talk that we gave last year. And I like to throw the abstracts into the slides so that people coming along later if they just only have the slides they have a little bit of an overview of the talk. But what we're going to talk about today is this outline. So I'm going to review some of the rest API concepts that we introduced last year. And then the main portion of this talk is going to be talking about what we've accomplished in the last year and one of the big features that I'll talk about is resource model. How that relates to the API. And then we're going to give you a bunch of demos so we have new APIs in several different areas. We're going to talk about those APIs and give demos for power management for camera and for serial port. And then one thing that is also pretty neat that we've done in the last years we've extended the API to be able to use it from several different test frameworks We'll show you a bunch of different ways that you can run tests using the API from lava from Fuego and from the robot test framework and even standalone just on the board or manually. And then we'll wrap up by giving some future directions. So let me review just the kind of the overall concept and introduce the problem statement so what problem are we trying to solve. Well, the big problem in our opinion is that people can't share tests with each other. And there's a lot of different tests that people run in their own labs, but there's no standardized way of accessing the hardware in a board farm lab. So there are a lot of different test rate frameworks. There's not a kernel CI or robot test for framework of things I mentioned earlier. And there are a few board farm frameworks frameworks that exist for managing boards, but there's no standardized way between all these different labs and test frameworks to run tests. Well, the problem is that every farm board farm implements their test infrastructure differently. So what will usually happen is you'll get a bunch of boards in your lab, or in your farm and you'll then you'll get some hardware to do things like power control, or usb switching or SDMuxing or whatever. And, and you'll cobble together the hardware that you can get that's available. And then it's not a super hard thing but you'll write a bunch of custom strips for control of that hardware and for data collection. And the problem is that it's different for every lab, right? So the test, a test written for one lab was not going to work in another lab. The end result of this is bad. It's that nobody can share their tests. And so we're missing out on kind of the open source effect of in the QA area of Linux development. So our solution is to create a standard method to access a board farm and specifically this will allow technology so that technologies can evolve separately from the interface to the farm. So that interface creates an abstraction layer. So if you write to the single interface, then behind the scenes, you know, drivers for different hardware can communicate. So the end result is that tests can be written to work that work in more than one lab. And that's kind of the main goal. The other thing that happens when you do this is that now test frameworks can actually work with more than one lab. Right. So not just the test but the frameworks themselves. So this is a high level diagram showing a bunch of different test frameworks and if they all use the same API they can talk to boards. And these boards can reside, hopefully anywhere in the world, and they can reside in different types of labs. And, or, or in the same lab. So it completely abstracts away that issue. And then if we drill down a little bit further. We see that there are not just the board that you can control, but there's hardware in the lab. So some of the API's have to do with manipulating or accessing the board. And, but a lot of the API's especially the ones we've added in the last year, have to do with controlling other hardware that you might find in the lab like a power measurement, or audio video capture, or controlling the end points for like Siriport or usb connections in the lab where we can manage not just the endpoint on the device under test but the endpoint in another on another piece of hardware in the lab. So that's, that's the overall kind of concepts. The rest API itself consists of kind of two major parts. We have a web based rest API. So that is, and then a command line interface the rest API is based on HTTPS and JSON. So, and the system that we're using is actually an extension to lava or Django. So lava itself is a Django app, and Django has popularized a particular style of API for manipulating objects on a web server. And so we just extended that existing API. And the great thing about it is that you can actually perform this API using only two command line utilities curl and JQ so JQ is command line utility that that can manipulate and access fields and JSON string and so JQ stands for JSON query, I believe. So you can actually perform all of the API, if you'd like, by just typing stuff straight on the command line using those two commands. But it's much more convenient if you use one of the command line tools that we've created so we have a command line tool. It performs the same operations as the rest API but it wraps up the parameters and the arguments for you. And, and so this is actually suitable for both human use and for automated use. And so, now, so that's, that's kind of in the review. Now this is the main part of the talk what has happened since last year we've actually accomplished quite a bit. And I'm going to talk about each of these we've added more implementations. We've added something called the resource model. I'll explain that in just a bit. And then associated with that resource model we've kind of added a generic capture API model. And then we went ahead and we added APIs for new resource types. So we have APIs for power measurement image and video capture and serial port receive and transmit. And then we have a couple of other things, kind of miscellaneous extensions that I'll talk about when we get to that. So the first thing is implementations. And the original implementation that we showed last year was lava server that is based on Django. And this is the same lava server that's used by Kernel CI and, and many other projects. So this server is actually in production now times this has a service called embedded board farm, and they are providing production services to their clients and customers. And so the client for that is actually available now and again repository. It's a shell based client using curl and jq. And the the so but we've extended that implementation obviously with the new APIs that we've produced this year. The other thing that we have available is something called lab control, and it's a server and client implementation. So the lab control server is a plain CGI script it's not written to any particular framework. And LC, which is the command line client for that is a Python client. So we've got a client in Python and got a client in shell script. And this Python client uses the Python request module. So it's going to format the commands or the wire protocol, according to what kind of the defaults that request module uses. And the source is available now, but it should be considered kind of alpha quality it's not production grade yet, but you can get it and play around with it and see how it works. So, one of the reasons for doing multiple implementations is that you can find incompatibilities or you can, you can shake out the API right to see, see how easy it is to implement but also kind of what are some of the issues if you, you know, if you do it in different languages that type of thing. So our ultimate goal of course is to use any client that's written to the API and have it work effectively with any of the servers. And so we've done a bit of compatibility testing between the clients that we have now and the servers we have now, and several of the API is work pretty well. There's a couple of incompatibilities, but, but we're kind of working on those. So the incompatibilities are in some of the new API is where places have diverged but the core APIs are all working just fine. One of the things we found was that in using the Python request module showed that there were some issues with the API definitions, the specs that we had last year so the curl command line tool and the request module from in Python actually perform the same operations with different form encodings, right so you have to, in order to be compatible you have to make sure that the wire protocol matches exactly. So you can't be passing stuff in different formats. So that was something we discovered and started fixing and the other thing with these separate implementations is that the other labs have other have API is that the other lab does not support for so for example, the EVF lab, the Times this lab has API is for storage management and they have some, some other API is for USB, and a couple of the things that they're using for their production testing. And their customers that the Sony lab that runs lab control does not have so then the Sony lab has serial receive transmit. API is so there are a couple of differences in the implementation but the core that we discussed last year and, and the intent is to have the API supported uniformly throughout. And then the next thing I want to talk about is the resource model. This is probably the most important thing that we kind of worked on this last year so previously all operations that you performed in the API were relative to a board you can see in that top line example that there's a board in the URL string. And so you would have a board and action and maybe some parameters in this case power on. But we introduced a new resource model, so that besides just being able to address boards, you can address resources in the lab so first you get a resource that is associated with the operation type that you're going to perform. And then you perform operations with that resource instead of the board so for instance, in this example there's a camera, and we can do a start capture and we address that to the camera not to the board. And so there's a new API to retrieve that resource assignment that is board based, and then you use that resource in subsequent calls to to perform operations. So the reason that this is good is that it's much more flexible than just the board specific model. So you can have resources in the lab that where you can have more than one of the resource or of the particular resource type assigned to a board. So you might have multiple serial ports on the board, and multiple endpoints in the lab correspond to the, you know, opposite endpoint of those, or you might have multiple probe points on a board for power measurement so instead of just one power measurement. You could be taking power measurement from several locations on the board. And so you need to be able to address more than just one of a particular resource type. Another thing is that so you can have multiple resources per board but you can also have multiple boards per resource so it's very common in a lot of labs to have a power control device that can that runs or controls multiple boards and there may be operations that you want to perform that are not board specific like a bank power switch or something like that. So also the resource model gives us a lot more flexibility in the future for doing something that I call dynamic multiplexing. And that is the ability to take a resource that is in the lab and use it with more than one board, you know, serially or sequentially so you can control the resource assignment at runtime. So one example of that that you might think of is like a video switcher. If you can control a video switcher from the command line, you can have it work with one board for a particular test and then switch it and use it with another board. And so there's, there's other pieces of hardware that are capable of doing that so makes it so that you can use hardware with more than one board. I'm not sure that we have today but the resource model kind of anticipates it and should support it without much problem. So this is actually kind of an even more detailed diagram showing that relationship that I said we're so you might have multiple resources of the same type associated with the board so audio video capture you might have multiple video outputs, or power you might have multiple probes. And so all of those are available and you can access using the rest API and in particular the new resource model. So the supported resource types that we have right now that we worked on the last year we worked on power measurement and camera and serial. But we think that other other resource types are obvious ones that we would like to work on so we'd like to work on canvas. We've already purchased some hardware for that and and USB. I mentioned that times this already has a few apis in their system that are working on this so. So, so that's the resource model so another kind of related to that is what I call the generic capture API model. So, what we wanted to do is come up with an abstraction that can be used for lots of different resource types and. Okay, this is kind of the equivalent of in Unix is the file, everything is a file in in in this system. The rest API everything is captured not everything but but you can apply these same verbs to a lot of different resource types. So start capture stop capture get data and delete and those are kind of the basic verbs that you use for any particular resource where you're doing data collection. And so, in looking at how these are used with different resources so in power measurement that's basically you have start capture stop get data and delete. So in order to extend the API for other other systems, or other resource types, we determined that there were extra parameters that might be nice to have, or convenience functions that were nice. And so we added a couple of other APIs that you see there in red. So for image and video capture, we can capture a still image. We can also we added start capture with a duration so if you want to. You can come back and manually do a stop capture you just say well I want 60 seconds of video. You can do that and then get ref is a call that will return a URL for a video file without you having to download it and you can download it if you want. But get ref allows you to just leave it on the server and you can reference it during the test that way. And then for serial port we also added a couple of things. And so with well one for when you're dealing with a lot of buses, you're going to have to do some configuration so we added a second thing API and that is used for configuring things like the body rate or transmitter receive mode. And then for serial port receive. So the DUT is the receivers the device under test is a receiver so the lab resources a transmitter so all these other APIs have to do with capture. This is actually the opposite one this is essentially the right operation which is put data. And so these are these are kind of the new API is now they fit into the resource model. And now just to wrap up here a couple other things that we did. One of the things that we added direct support for the API in fuego so last year when we did our demos fuego was using the ebf, which was the only client at the time. But I was using it via a wrapper called TTC. So, fuego already had support for for this. It was called a transport layer. But that was awkward for a couple of reasons so it's been modified fuego has been modified so now fuego can use specify a transport of either ebf or LC for a board, and the tests are performed using that API directly so there's less than the fuego docker container. This is mostly of interest to fuego users but the kind of shows that you can you can use this stuff be a wrappers but it's better to kind of integrate it more tightly. Okay, so some of the other things that we've done just a couple of miscellaneous features were that we've added support for recursive directory copy. So last year we could copy individual files to and from the device under test. So we can copy an entire directory, either to or from upload or download. Also, in terms of debugging we've added a dash dash debug argument to ebf. And this is really handy. It generates a trace of the API request so that means you can get kind of detailed information about other HTTP request and response, the structures that were used and the encoding and stuff. And that's actually very very useful to see those data structures on the wire, especially when you're writing implementations that are supposed to be compatible so. So that's a new feature and then also in the test repository or in the ebf repository board farm rest API that we created a new directory that has sample tests so you can see. And in fact some of the tests that we'll show later on as demos are actually available you can look at those tests and see how they run and. I think will be helpful for people trying to experiment with the API. So now I'd like to go ahead and turn the time over to Harish to look at lab independent power measurement test so Harish go ahead. Thank you Tim. Lab independent power measurement test. This test captures power measurement data for a device while running some workload on the device workload could be any CPU memory or file system makes extensive operation analyzes the collected data and reports is a device under test duty is connected to a lab resource, the power measurement lab resource in this example duty is a Raspberry Pi board and lab resource is ACME power measurement port. Before we close the binding of duty with power measurement resource it knows which Raspberry Pi is connected with which ACME power measurement port or which Beagle bone black is connected with which Sony debug port. Before going into detail of power measurement test and recipe I let's have a look at our measurement demo. Here we are going to see three different ways of running power measurement test test execution directly on the device running it from a remote system and test running as lower framework test job. So here is the first example of direct test execution on the device. For this, we will be accessing device console through time says ebf serve interface ebf stands for embedded vote form. This is the device. The black portion of the screen is a device console. Here we are running our measurement test which is already deployed on the device test is running countries performing some workload stress test CPU stress test on the device. Power measurement reading max power consumed is 2.141 watt, which is below threshold configured on 2.5 volt and over test fast. Next running the same test script on a remote computer. So this computer which is running test script is in India whereas our device server and lab resources are in a lab in Pittsburgh. All the interaction between test script and lab is happening over the rest API. There's no need for VPN on any other connectivity. So it ran the test and this time max power used is 2.133 watt. Good below the threshold configured. So here we are running a test through lava framework which takes care of test scheduling and result collection. We manually curated of lava test job email which would run our power measurement desk script on the three by three device which we copied on the screen. So I'm meeting the desktop. Device allocation for test execution device got executed test job is running. We could see lava desktop logs coming on the screen. Our test executed and this time max power is low 2.102 watts. And if we go to lava test results. So this was a short demo of power measurement test. Let's get under the hood and see how rest API is working. So here is a snippet of a power measurement test scripts. The statements which are highlighted CLI tools command which involves power measurement rest API is the test starts by grabbing a power measurement resource. Then it starts power measurement data logging by running power measurement start commands. Then it runs some workload CPU stress test on the device once the stress testing is done it stops the data logging. It grabs the data it computes the max power deletes the data from the server and finally compares the max power with a threshold to pass or fail the test. These are the CLI tool commands for power measurement operations. Get resource start capture, stop capture, get data and delete data. In the CLI command this test client is lab dependent. For times lab it would be EVF for Sony it would be LC. The CLI commands and the underlying rest APIs and their specification are same. So if you want to run the same test on different lab could read this test client value from a test config file or an environment variable and then the same test would run on different lab to different infrastructures. These are the rest API is used by the CLI tool and ever test. In these APIs they return a JSON object in the output with three properties result data and message result is either success or fail in case any operation fails failure region is returned as message value. Data value is API specific or get resource it returns power measurement resource ID. Data data returns are unique token which represent the current power measurement data logging which is taking place. And in case of get data it returns a CSV string with power measurement readings, where each row has timestamp, voltage and current readings. Next is our lab independent camera test. This video records video of a device while it reboots in the lab. Our device under test or duty is associated with two lab resources this time a camera resource and a power controller resource and duty knows the binding of these resources with camera camera resource takes video of the device whereas power controller is responsible for rebooting the board. This is the short demo of camera test. So again we are logging into this web interface here. This is the device use for camera test. As we see here device is currently powered on device console is operational. You get live video streaming device already is on and desktop screen is visible on the monitor attached. Now we are running this camera test from a remote system. The first argument is the device name and second argument is for how long you would like to do video recording. So we are rebooting the code and in the terminal it prints out you are a lot of the recorded video. Let's play this video. So as you see here, initially devices on already is growing. Now devices off it's going through reboot procedure. The device is booting up. You could see boot logs coming on the screen. An important point is this camera resource which we are using for video recording we could also use this same resource for capturing still images of the device as well. Booting is almost done. Booting sequence is over. So this was a short demo. This is the camera test script. Again here the highlighted statements are CLI commands which uses camera APIs, camera rest APIs. This starts with grabbing a camera resource here. It starts recording a video for a preconfigured duration reboots the board. The duration is over. It prints out URL of the recorded video on the terminal. These are the CLI commands for camera operations, get resource capture, start capture and get reference. Get resource is similar to power measurement. Get resource resource only difference here is instead of power measurement resource type is camera. Capture takes a still image of the device and returns URL of the captured image with minus o argument that image is downloaded on the local system which is running the CLI tool. Start capture or start recording a 10 second video with minus T argument we could record video of any duration less or greater than the 10 second. Get reference it returns URL of recorded video with minus o argument similar to capture it downloads the video on the local system. These are the rest APIs for camera. Again, these APIs also return JSON object with result data and message properties. Data here has value of camera the source ID capture has URL of the still image. Start capture returns JSON with token property which has unique video ID of the current video capture which is taking place and get reference returns URL of the recorded video as data value. With this I hand over to Tim again to give us insight of serial test. Okay, thanks. So, in the Sony lab we've implemented serial port testing and this shows the diagram we've got a device under test and it's connected with a serial cable from one of its you arts out to a lab resource which is capturing it. So the steps are pretty straightforward. The first thing is to configure the lab resource. In this case this is a transmit test that I'm going to show so would be configured lab resource to receive and then set the body rate on that side using the rest API. And then we use local commands on the device under test to set the set it into that end in the transmit mode and with the body rate. And then we capture and we, or we start the capturing start receiving on this lab side, and then sending on the DUT side and then we end the capture collect the data and compare it. And that's the basics of the test. So, let's go ahead and show you I'll show you some this test running from Fuego. In the Fuego I've got a board. It's a big one black it's called LC BBB, and I'm going to start this board farm serial transmit test. So I'll go ahead and start the build on that and this is using the Jenkins interface that's the Fuego is kind of preferred user interface. And then so we get some setup stuff out of the way and then we start measuring stuff so notice that we've found the resource name for that you are one which is serial GE. And then we're going to test a bunch of different baud rates and we're going to do the same steps for each baud rate. So we're going to configure the baud rate. And then capture the data, transmit the data from the DUT, stop the capture, and then retrieve the capture data and then compare it and if everything compares okay then, then we call that a pass. We have operations to download the data from the server and then look at the overall set of results and see we get all past results for all these different baud rates. That is a pass on that hardware test. So, and then the same thing, looking at what it looks like in practice in terms of the APIs, you see highlighted lines here, showing that in order to do the set config, we actually send some formatted to the set config command and that allows us to have an art kind of arbitrary attributes corresponding to whatever attributes apply to a particular resource. So for a serial endpoint you'd have a baud rate, maybe things like X on X off or stuff like that. For other resources you might have other parameters that you want to set. And then the same types of patterns you've seen with the other tests we get a token when we start to capture, we can use that, use that token to actually stop the capture and retrieve the data. And then at the bottom there you see us comparing the send data and their received data to validate that the transmission went properly. So in terms of the CLI command tools, again, very, very similar. The only thing I kind of want to point out that's different here is on the Git resource, notice that there's an extra parameter called the feature. So and this has to do with the being able to identify which endpoint in the lab you're you're referring to so the Beaglebone black actually has five different you arts. And so you have to identify to the lab will which which serial connection are you are you referring to. In this case we're referring to you aren't one, which is on a certain set of pins. And so we have to find the lab endpoint that that has the connection to that you are one so you can actually have multiple serial ports defined in your lab and connected up and test them independently. Other than that the API is basically the same as as the other capture so the capture model has actually turned out to be fairly generic generic enough to use with multiple different resource types. With with tweaks here and there as you see as we you know as we flesh out the API. And then looking at the API details the same type of thing so we have a bunch of paths that we pass as part of the URL. Again the only difference on the grid resources the addition of a feature at the end of that path and then we're returning JSON. Same thing result data and message in the case of failure. So that's the serial API test. And the other thing that we'd like to talk about that we're pretty excited about actually is that we have run this API or use this API from multiple different frameworks so you've already seen an example of power management test running standalone running remotely. In Lava and showed the serial test in Fuego we want to show the same power management test with two other frameworks so we'll show you the robot test framework and then Fuego running the power measurement test. And so this is kind of the Rosetta stone we're going to show you some code from each of these and there's lots of similarities but it also helps you kind of understand. So how do we to these systems where where would you put the call outs to the to the client and stuff like that so let's go ahead and I think we're starting with a demo right for robot test framework. Robot framework is a general purpose framework for automation. It's Python based. It is installed with Python pip command you could simply run pip install robot framework to install it on your system. This is a power measurement test return for robot framework. It uses same CLI commands and power measurement rest API as the previous test we showed earlier in the talk. In this test we first import process module for running CLI tool commands then there are a bunch of variables which are defined here, and then under test cases section and there is only one test case get power measurement test. And all these test steps are same as the shell script based power measurement test to grab a power measurement test resource starts power measurement. Run some workload command on the device once the command execution is always stops the data login grab the data. If it computes max power, then there is a verification point should be true where it compares max power with threshold to pass off in the test case, and finally it leads the data from the server. Now let's look at the video of this robot framework testing execution. We are running this robot test script on a remote system we could run it from terminal or from the UI here we are running it on typing on the terminal with robot command. It's executing the test. It prints the result summary on the console also generate a nice HTML report. Let's have a look at this report. So robot framework consider each script as a suite so you click on the suite and then it shows the individual test cases also it had only one test case. If you click on the test case it shrills down and shows the test case blocks. So that was a very short demo of Robert finger power measurement test. Now it's over to Tim. Okay, so now I'd like to show the same thing running in Fuego a power measurement test using the rest API. So we're going to run a video here. Once again we're in the Jenkins user interface I'm going to be running it on the LC BBB board, and go ahead and start the start the test. And we can go look at the console for the test and see that that first line there we've got the ACME resource that we determined was associated with this. We're going to start measuring the power at this point and start a stress workload. So let's go back and look at the historical data for this test we see that we have a reference power usage of 2.2 watts and none of the previous tests have gone over that although that that build that test run almost did. But we'll go and look at the test and see that it's finishing up the stress operation. We're gathering the power measurement data that was collected at the server. And the first few seconds are a little bit less current as we get into the workload you can see the current goes up from about 250 to 380. So, but the overall maximum power used is still, let's see 2.17 watts. And so that was a test that passed. In Fuego, we're going to be doing the same types of operations with the client that we did in other systems. Fuego tests are written in shell script language and so it's very similar. Just a couple of differences that might be of note here is that when we, the Fuego has the parser in a separate module. So when we get the power measurement data, we actually emit that into the test log with some delimiters, start power data and end power data. So that makes it easier for the parser to extract just the power data. And then performance analysis against, you know, relative to the threshold and calculate the max power. So but that's what it looks like for Fuego power management test. There we've showed you several different power measurement tests and lava robot framework in Fuego. What we'd like to do now is this is, we'd like to promote the use of the API and these implementations. So the implementations are available now, you can go out and lay around with them. If you have any questions about them, please contact me or Harish. And there's actually, I think a mailing list that you can get on, if you're interested. Some of the things that we want to do is we want to add more API. So we've done a lot, but we're, we have more stuff that we want to do. So can bus is next on the list. I kind of mentioned that earlier. We expect to be able to use the same APIs second fig capture and put for can bus operations so we can do hardware testing. We'd like to do USB connect and disconnect testing. So we have hardware that can can do some of that. That is useful for testing like the robustness of mass storage or connections to other devices that are connected on the USB bus. But the question that we have for you is what APIs would you like to see. So we, we'd like to hear about what types of hardware testing you're doing in your lab, how you're going about that and whether or not this API fits that model. Find more use cases and, and if we need to change the API, we can still do that we're in early stages here. But we'd like to see if this model works and then continue to extend it for use cases for throughout the industry. The other thing is we'd like to add more clients and client examples so LTP is a Linux test project they have a serial port test. That's a hardware test and we'd like to see if we can run that in conjunction with our, our clients. And, and then also we need to upstream our changes from the ebf production server to the lava upstream project. And then we would, the big thing here is to use this in more production testing so we're going to find out more about the API's and use cases as we do more real world testing. And that will help refine the API's. And then, last thing here. So we have defined an API now that is between essentially the test or the test framework, and the lab server, the or the board farm server, and that is a good place to have an API that allows us to to standardize one aspect of the entire test stack. Another aspect or another part of the stack is the interface between the server and the lab hardware. So, we'd like to establish standards that would allow us to drop lab hardware into existing labs. And so one example of that right is there's something called a Sony debug board that does power measurement and it does power control and support capture. It'd be nice if I could so I have it all configured so it runs in my Sony lab but it'd be nice if I could just ship one of those as a device out to someone else's lab the time since lab and have them install and just drop into place. So essentially what we're talking about would be like the hardware drivers for resources to make that a standard as well. So that's not something we have today but that's something that we view as important long term. And so we'll be working on that as well, trying to consolidate that. So, with that, I think we are done with the presentation so this is our update of our automated board farm REST APIs, and we look forward to hearing feedback and comments from you that's the reason we come to these conferences to kind of tell you what we've done and get more information and we will be here. I should be there in person if I'm not something has gone horribly wrong and will be available in chat and we can talk to you with any questions or comments you have. So with that I'll say thank you very much for your time and we'll see you around.