 Today I'm here to talk about teaching your test framework or your test runner to speak Lava. So how many people are already using Lava and have been quite familiar with it? How many people have just heard about Lava, they're interested in what it is and trying to use it? So for those of you who are very new to Lava, I'm going to be going into a little bit more depth than you're probably quite ready to grasp. So there's a number of prior ELC presentations on, you know, introductions to Lava and actually Jan Simone Mueller just gave a talk yesterday that gives you a good overview of it and things like that. But I will do my best to not dive too deep to lose you. So most of us already have tests, right? So we're coming to Lava with tests that already exist. You already have a team that's been writing tests in some kind of a test framework of some kind. So wouldn't it be great if Lava could auto discover your tests just like PyTest or something like that can do, right? Just point it at a folder and say PyTest and it goes. Wouldn't it be great if you could just run the same commands in your CI system or in Lava as your developers are already running, right? You don't have to do any special incantations for Lava, you just run the same thing. So this is, you know, to keep it simple and I don't want to put the words stupid in there so I changed the smarty pants, but that's a, you know, common approach in software. Also you don't really want to rewrite your tests just to make them work for Lava. So, you know, don't repeat yourself, don't add extra work. And also, you know, the traditional approach in Lava world is to do post-processing of your results, post-process parsing. Wouldn't it be great if you didn't have to do that? So that's what I'm talking about today. So let's just go a little bit into the life cycle of a test case in Lava. By this, what I mean is what shows up on the dashboard in Lava and why. Why does it show up as a test case? So the reason is that there's these special signals. So number one, this Lava start, or Lava signal start TC ping test, right? So once you have started a test run, Lava does a bunch of other things. I'm not going to go into you that before the test starts running, but once you have started a test run, Lava is only observing standard out period. That's what it is doing unless you hit a timeout or something like that. So it is literally looking for the string Lava signal start TC and then whatever the test case name is. And then you've got some stuff. And then you've got Lava signal end TC, then it knows the test case is done. And then it knows what the result is by Lava signal test case, test case ID, so on. So in this case, this test failed, but you can see at the very, very top, the plus Lava test case ping test, that's using a special incantation for Lava, which is where we're trying to get away from. So there's a little bit of extra stuff, sorry. So let's look at what can we do, right? This was my aha moment when I realized this is what Lava is looking for. Why don't I just make my test runner, my test case runner, just emit this stuff. So I've got pseudocode that's sort of Python-ish on the left. So why don't I just go ahead and print out, hey, Lava signal start TC, whatever. And then do my test, and then Lava, et cetera. Why don't I just do that, see what happens. And the result is going to be exactly the same, right? Because again, that stuff is going to standard out, and Lava is just observing standard out. It doesn't know whether Lava test case, the script, wrote it, or whether your runner ran it. There's a little bit of extra stuff that's available. It's not super commonly used, but you can have a numeric measurement. So in this case, just purely a number. You could have pi emitted. And you can also have units, measurement with units, like velocity. And so that just adds a couple extra clauses to that last statement, the Lava signal test case line. So this is another idea that I had, is like, oh, now I can actually have my test case runner emit those things as well. And now I start seeing results showing up even with measurements and units. So the first thing that my team looked at is PyTest. We're all really, really comfortable with Python. And so we just naturally went straight to PyTest. And we wanted to write all of our tests in that. So I gave a talk at ELC in Portland last year about running a rolling release with open embedded Yachtile project-based thing. And I disclosed that I had been doing this stuff with PyTest, and I talked about writing a PyTest plug-in. For various reasons, I haven't quite gotten that done yet. So I'm just going to go ahead and show you what we actually used. So what we really literally did is in the same folder where your tests exist, where you're going to call, you're going to say PyTest, go to this folder. In that folder, you put this special file called ConfTestPy. So PyTest is looking for this specific file name. And in this case, I need to convert the results from PyTest, which was this PyTest outcome, past, or skipped, or x-failed, right? So in lava, there is no such thing as an x-fail or a skip condition. Sorry, okay, then I need to change this. Thank you very much. I meant to double-check that, and I didn't fact-check myself. So I need that one line, you should change that to skip. Okay, there is no x-failed. So x-failed should just be a pass, okay? I think they might not have been a skip earlier, or anyway, it doesn't matter. My bad, okay. So the other thing that you need in this case is where are those actual signals going to get emitted from? So there's a special function that we're overriding in this case. So it's a report type of function or type of object, and basically you can see this is quite simple. So we're looking at what is the status of, or where are we in the PyTest run? So this is traditional unit testing. It's got set up and tear down clauses. So in the setup case, we want to emit that start signal, and then when it's actually in call, we're going to figure out what the result is at the end of everything. And then we're going to emit the end signal and the test case ID and result. So this is the simple case where there's no measurement in units. I have an appendix where I show how we would do it with units if we have time. So this is what my test definitions look like before and after this change that I've done. So you could have just run, I picked Python 6 just because it happens, or just because it happens to be a module that uses PyTest. So you could have just run lava test case, give it a name, Python 3.6 tests, and then just run a shell command, PyTest. So really, there's no difference from what we're calling. On the right, we're just going to say call PyTest directly. So the difference is if we look at what the output of the first run would be, we get all of the tests, everything that PyTest was going to run in one big blob, and we get the result for the one big blob, and that's it. PyTest happens to, by default, output these dots for passes and S for skips and F for fail. But you can see it's still got the same pattern, right? Over and over again of start Tc and Tc test case. So what happens when I use that ConfTestPy that I just showed you? Literally copy and paste that whole file into your area and run it. What we see is that it's going to pick up each individual test case. So I've abbreviated this because it's hundreds of lines. But you can see now there's actually a test case named AdMediclass that passed. And then there's a test case named AdMediclass nested that failed. So why do we care? The reason I care is I tend to do a lot of looking at my results through the web UI. And so when you look at the results tab or the results part of your lava dashboard, in the first case, all I'm going to see is Python 3.6 tests. My entire test suite failed. If I click on that, it just shows me the top of the test suite or shows me the whole test suite failed and that's it. I don't really have much information. With the ConfTestPy that we added, I've got 20 pages of it with 10 on each page. 20 pages worth of results of test cases. And it actually shows me that the only thing that failed was this AdMediclass nested. And if I click on that now, it's actually going to take me straight to that part of the log or the output and show me what that was, okay? So to me, this is a tremendous benefit for extremely little effort. And I don't have to do any post-processing. And if any of these tests fail that some weird random way that ended up making lava crash out, I don't know how that would happen. But if it did, I might still have already captured some results. So for this talk, I decided, let me go look at some more test runners and see what else can I do. Because this is a very, very simple technique. All I need to do is have a set up and tear down change. So BATS, BASH Automated Testing System. You get to write all your tests in BASH. So basically, I just added this dash L. So I added a lava option here. And again, you can just copy and paste these. And this was a DFT on top of Master as of a couple of days ago for BATS Core. I need to give some nice help info. So you emit the signals to lava, blah, blah. I need to set up a flag and make it cleared before I start running. I need to actually do something when that lava flag is being set. So I just set this variable to 1. I check if that variable exists. And then I change the executor to this BATS format lava stream. And I just copied that straight from the BATS format tap stream and just changed it a little bit. So here's this BATS format lava stream. And I just did a diff between what I did and the BATS format lava stream on a tap stream. So I added in result measurements and units. I have not actually implemented measurement and unit stuff yet to see if it works, but I put it in here anyway. And then at the beginning of the run, I need to do set up. Because this is where I'm going to actually emit the signal. So I needed to add that in. Again, I need to convert my results. And again here, I should have made this skip, but forgive me for that. And again, we'd have to change this to fail. It turns out that the test case names are going to have spaces and things like that in it. And lava needs that to be converted to something else. I just made them all an underscore because that's a pretty common convention. So that's what this is doing. And then here, this buffer is sort of like printf, basically. It's just a wrapper for another function. And so here, I just emit the start of the test case. Tear down. Again, I need to manage this name, escape it. And I just buffer out the test signal end. And then I do a little bit of testing to see whether I've got a measurement or not. And then emit the test case signal. So here's just a very basic test definition. So again, on the left, you can still just run this pretty simply. In this case, the dash t is for tap output. It's a little bit easier to read. And over here, I've added my new L flag that I've created. So again, if you look at the original run, you're just going to get a whole bunch of data. There's 70 different test cases here that ran. So there's a bunch of stuff I'm skipping. But I just showed the one thing that failed. And so the whole test run is going to fail because of that. But when I add the dash L change, now again, I'm going to see every single test case. And so I'm going to see the output printed even when no final new line test case and so on. And I'm still going to be able to catch when an individual test case fails. And I skipped a whole bunch of stuff because it's too much to put on the screen. So you're going to see this pattern, right? I'm just going to drill this into you, but forgive me. So for the whole run, I've got just this whole test suite that ran. My result is fail. If I click on that, it's going to take me to the entire log, which is hundreds and hundreds of lines long. And I've got to go searching for where the hell the failure was. You can grab it. You can do whatever you want to do with it, right? But ultimately, it's just frustrating to me to have to do that. I want to get to it as fast as I can. So what happens when I actually just add these very few simple changes to the runner? All of a sudden, I get seven pages of results with 10 results per page. And I get all the way down to seeing that, hey, these are the two test cases that failed. And now I can click on these, and I can get straight to that part of the log immediately and see what the failure was, right? So what else can I do? So Jan Simone Mueller gave a very, very excellent introduction to P-Test, part of the YAKTA Project open embedded test suite yesterday. So this lets you test individual packages, and you basically do the equivalent of make check or make test of whatever your software is. Currently within Lava, there's some Python scripts that do post-processing of P-Test results. And they essentially will give you the same result that I'm going to show, but it's post-processed versus processed at the same time. So the first thing I needed to do to make this work is modify the P-Test runner to test case. It's C++, or C, I mean code. So I needed a flag. So I'm just going to create this flag in a way so we can expand it later in case we need more. I need to include that. I need to add a new test runner or a new option. There was already an L option for list, and so I went with capital L. I need to add to these options that are available. I need to zero out those flags. I need to add the capital L to the getopt and no colon after it because it doesn't take any values. I need to add this case for the check and for the L, and we're just going to bit or that value with a lot of signal enable, that flag. And in this case, I printed it out just so you can see and you know that you enabled that. In the utils.c, this is where the actual emission of the results happens, so I need to include that file. Turns out I need to get what the P-Test name is, and that was what I did here. So this directory would give me the path to the test, and I wanted the name of the exact test case. So I get the results, make an array for that. We know it's going to be five characters because we've only got pass, fail, and skip. Here I knew it was skip for some reason, ignored it elsewhere, and then we just check with just a regular and is that bit set? And if it is, we're going to emit the start signal. So then here we're going to check if that exists. We're going to print out whether it's a pass or a fail. Didn't add a, I guess I should modify this to add a skip option. This is kind of the tear down, and so now we're going to print out the end test case and the test case ID stuff. So all of this code is there, it's pretty obvious, I'm not going to go into all of it, but so in this case, I am the maintainer for the Metapurl layer in Open Embedded, and so I wanted to do a bunch of tests on some Debian named Perl modules. So on the left we've got the way I could have done it, on the right we've got this very similar bit one modification running that capital L in there. Again, if I do the regular run in this case, it's going to start the test case runner, and it's going to start the first of all those things I had on the command line, or if I ran it with no options, it would run all the test cases available. And as Yon Simone Miller said, in yesterday's talk you end up with five megabytes or more of log that you have to go through and try to figure out what the results are. This at least breaks it up a little bit. And at the end of it all I've got hundreds and hundreds of lines of output I skipped, and then the last tests that ran happened to be lib digest HMAC Perl, and I'd get an overall result. If I use my change again, you're seeing this pattern. So now it actually says that the test case was this lib capture tiny Perl. That was the first in the list of P tests that was going to run. The last one happened to be, oops, anyway, this shouldn't be identical, my bad, but anyway, ran through a whole bunch of tests and each one of the test cases is going to run individually. So again, what I would normally see is just one great big blob pass, right? The entire P test run of everything that ran, what's the result? How do you drill down into that and see any differences? Pretty hard. So with my change, I actually now see each individual P test suite that ran and I see the result for that. In doing this, I actually caught that the P test runner module that I modified personally in the past needs a little help because not all of these passed 100%, I don't think. So there's something missing there. But so just in summary, I think you can see this is really a very, very simple concept. I mean, I'm surprised I could spend this much time talking about it, but I wanted to go over at least a few options or a few examples of how you could do this. So I think it really improves the dashboard experience, what you can get out of it. It opens up the ability to do charting and queries, which I didn't really have time to prove and show you today, but there's a lot more you could do with the dashboard because of this. I think it's way easier to find the cause of test failures. Pi test, it works pretty well. The output gets a bit cluttered because it's really just showing you whatever dot, whatever and so on. The output leaves a little bit to be desired. I do have an actual Pi test plugin, which I'm gonna call Pi test lava, I believe, as long as the lawyers agree with me. That's probably gonna be hosted on kernelci.org, but I don't know yet. The real reason that it's held up is because the measurement decorator stuff that I've got in the appendix isn't all fully implemented yet. And if we have time, I'll go to that. Bats, I was surprised how well that worked. I mean, it was just easy. It'd be nice though, because of the way they did it, the tap output isn't there in the way I've got it running and it's not easy to enable it. And so right now the dash L and dash T options, the lava and tap output are exclusive. And I haven't figured out exactly, I haven't tried the measurement and unit's functionality yet. It doesn't seem like it's that commonly used of a framework, so that's something to consider. For P test, what I caught here is that each test case in P test parlance should really be like a test set or test suite or some other term in lava parlance. And we should actually break it down even further, so we need more work there. I think we would get a lot better results, at least for the Perl examples, each of the T slash whatever should be broken down. And this showed me that some failures might be hidden if you are not using, so in P test the runner is a script called, or the thing that actually is evoked for each test case is a script called run P test, which is usually bash. And I actually went back and caught the fact that I had not set X, right? I had not set this up to export the result of every single command that was running, and that's why I have hidden failures. So I'm gonna go fix that when I get back. So I think going forward, you know, I think all of you, I've literally given you everything, I've given you all the code to make this work. I think we should all go out there and enable more test runners. You know, whatever your favorite test runner is, let's figure out how to make that work. So I just wanna thank everybody for this opportunity and thank some team members that really helped me out a lot in this work. And I'll go for questions now or we can show the measurement decorator thing, but yeah. So I think I am 100% for tap parsing in lava. In fact, I would like to have, rather than just the CSV and YAML outputs, I'd like to be able to get J unit or tap output from lava. So that's a great thing, I think absolutely. Not all of the tests that I'm gonna be running here, not all of them have tap output, right? Yeah, but I think you're absolutely right and that's what I looked at was, could I figure out how to add that all into lava easily and how quickly could we get that in and merge it as a community versus this was down and dirty and it worked, right? So I'm showing you a shortcut, it works. You're absolutely right, that's the better long-term solution, no doubt about it. There's others there as well, right? But that's an obvious one in lava, I think. Any other questions? Yes. Oh, sorry, can you come up to the mics? Sorry, I forgot to, this is recorded and lots of people watch these videos after the fact, I watch my own videos because I often forget what I worked on six months ago or a year ago, thank you. Question is short, is there any plans to do it with robot framework? To do that, what? Lava. Is there any plans to? To make such a cooperation with robot framework? Can, do you know robot framework? Oh, robot. Robot. Yeah, got it. Yeah, I used robot framework in the past. That one's a little bit more complicated because robot framework has an awful lot of ways of putting tests into it, which is why I didn't wanna try to show it here because showing what the tests actually were would be kind of difficult and then the output, it's got a lot of output options, but this is fairly trivial to do, so we could absolutely do that in robot framework. Yeah, and I've been a contributor to robot framework in the past, so we can absolutely look forward to doing that. Yeah, anything else? I'm just gonna go ahead and show just the measurement stuff just really quickly. So basically, we still need that lava result converter. We modify the conf test.py a little bit because we add in the ability to have this measurement text and we add in, in this case, the tests that we were running, we were running cyclic test and we're actually printing the units and the measurement at the same time. So this is just a slight modification to what I showed earlier. The one big difference here, which I didn't even, I missed the line, I didn't import, oh, I did it up here, so I have to import this measurement decorator. And so the measurement, and then the measurement decorator is just this measurement object. This is a little bit of a hack, but basically we just set up an array and in this array, we basically end up setting up just a measurement and the units, so your test case has to emit that or return that. This works for our case. It's a little bit hacky. It was not that easy to implement in the actual PyTest as a plugin. And there aren't that many tests that are actually emitting measurements and units, but this does work, but it does require you to actually change your test case runner or your test cases themselves. So I just didn't, I didn't really like this that much that we had to modify the test cases themselves and because I thought that kind of breaks with my original theme was don't modify your test cases for lava, just modify your test case runner or modify lava to parse it. But anyway, that's all I've got. If anybody's got any other questions about this or automated testing in general, I'll be around and happy to talk to you. Anybody, anybody? Anything else? Okay, thank you so much. Thank you.