 So, hello guys, so today let me talk a little about mashing up Fuego and Lava. So, let me quickly introduce myself. I'm Janssen Monmelo. Send me an email if you want some more info later about it. The slides are also uploaded. I do the releases and CI for AGL Automotive Grade Inox. If you haven't seen our demo in AW, take a quick look before you leave. So, well, if you just have one target and develop on it, yeah, it's perfectly fine to have everything on your desk, yeah, easily accessible to you. The problem which I have then is because, well, mine looks more like this, yeah. So, the picture is from our colleagues, free electrons, and that's just one of the drawers, yeah. Yeah, so you see it starts to get a little messy. And if I put all boards in one place, I'm not already there, right? So, that's the motivation behind that, yeah. And here, our guys from Linaro, they have, for example, here the, I forgot the name, panda board, right? Yeah, have here nice stacks of panda boards that are hooked up to Lava. So, when we looked at the frameworks for AGL, hmm, we were looking at, well, what's out there in the distribution space? There exists autotest, yeah. There are things like OpenQA, hmm. Yeah, that's a little bit graphic, but we are embedded. So, yeah, so OpenQA is more about taking screenshots and it can do all the desktop click stuff. So, yeah, probably not for us. Invented sports, yeah, we ended then up with, well, homebrew stuff, hmm, no. Okay, so we ended up with Lava. Lava does the board farm stuff very well, yeah. My absolute favorite. We also use Fuego. That is because it comes out of the LTSI project. Companies in AGL use LTSI kernels. So, we have that one also to play with. What we also see is we go from kind of, yeah, non-distributed specific environment to distributed. Yeah, that's a clear trend. Okay, Lava, I will skip that. We talked about that. Just a few points, basically my observations completely subjective. Very good to support board farms or multiple boards of the same type. Yeah, that's how it scales nicely. A lot of deployment mechanisms, even microcontrollers. The new slave worker setup is very slick. What I also found out about the initial setup is a little, it's a steep learning curve. And I read the documentation. Okay, now what do I have to do? You need to get into it a little and we'll get to that. Okay, so in contrast, what is Fuego? So, Fuego evolved out of the JTA project. So, the LTSI guys set up this project, JTA Jenkins Test Automation. So, it's Jenkins involved. We basically have a full Jenkins at hand and Fuego is basically a Jenkins prepackaged with abstraction scripts and a battery of tests. And all of this inside a container. Now, we can debate if the container is really the nicest thing. And on the one side, it's nice. On the other side, if you give that to your IT department, they say, and now, yeah, we want to use puppet Ansible and that doesn't really fit. We'll see. Maybe we will change it a little bit to use, for example, Jenkins drop builder underneath. And just add it to your existing Jenkins. That might be a better option. Anyway, so we have quite a few, I don't know the total number, but I think it's like 45 already existing test runs with complete parsing, with complete result parsing. So, that is kind of the big plus here. We have a large number of already predefined tests with complete reporting companies like Fujitsu, they added kind of reports over, I don't know how many iterations and yada, yada, yada. So, it's very good at the reporting because you have a full Jenkins where you can do plugins and I don't know. The setup compared to Lava, which even runs completely on a Raspberry Pi, here you have Jenkins and Java, so it's a little more heavy. And it evolved from a complete local lab setup. So, it has no notion about where my board is now at home. Yeah, and my Jenkins is over here. It accesses the boards usually over SSH when the board is up. It has no, well, no built-in way to deploy a file system, but the board doesn't know anything about it. It just assumes the developer made the SD card and powered up the board and then goes to Jenkins and does... Okay, so I said there's like, nice, but I cannot use it for our whatever code review automation. It's like I'm not going to press buttons here. Okay, so let's see what we can do. A, I want to distribute the tests and I want to aggregate the results. Now, let's say none of both tools could do everything. So let's see what we can do. So, distribute the tests. Another problem that we have is I don't want to hurt all the boards. Yeah, so for example, my colleague has board X, I have board Y. Let's have a setup where we can share those boards for testing. That works quite nicely. The new Lava has the so-called pipeline setup or V2 setup. And you can have a satellite lab, which is just a worker on a Raspberry, for example. And it connects to the master over an encrypted connection. Perfect. So I have my board at home. You have yours at home. We connect to the same master and we can now test multiple boards. The only requirement is here. Power needs to be controlled through a relay board. There are nice hats for the Raspberry and whatever. Or also nice relay boxes for network. That's perfectly fine. Serial. And what I usually do is an internal network just for the device under test. Which means basically that's my external network. This is basically firewall because I haven't set up any NAT. So it's completely decoupled. Actually, that could be a nice CT at home setup for testing. You just need a Raspberry, a relay board and a board to test it. Now aggregate the results. That's now a little tricky. We get the individual results basically usually the thumb up, thumb down. But I want more. I want to know what happened. I want the full log file. So I need a way to aggregate them, store them. And that's where kind of the Fuego part comes in place. Because it's built to run the tests, fetch the results, post process the whole thing. So that's where the strengths here are. So let's combine those two. So it's completely fresh. I've downloaded the code like one and a half day ago midnight. So it's not on GitHub, it's on Bitbucket. Bitbucket DL9PF. I should have put the link in anyway. So Fuego had a nice target setup link. That variable exists, executed. Oh, perfect. Oh, by the way, there are a few places. So where to add lava now? Fuego has the notion of a transport, SSH. Do I add it there? If we want to access the board, we bring it up. But we have no way to do some flow control in that case. Yeah. When do we know that we don't need that SSH connection anymore and so on? So I was like, hmm, nice idea would be completely transparent, but I have absolutely no knowledge how long a job runs. Okay, so target setup link is coupled with the drop. So I know start and end. Okay, I still had to add target tear down link because that was never thought of. So I just added that. And then we submit a drop to lava to bring up the board. Therefore, I need a lava drop template. I created a settings file for just environment variables for Fuego and then fill in the kernel I want and so on and construct my job for lava. It still uses SSH to connect to the board once it up. So that's a little, yeah. I want to fix that. Actually, I want to go from Fuego to the board and don't need that SSH thing because, yeah, it still assumes that the board is in my local network. So right now I have whatever VPN in between. And in my proof of concept here, it's using a hacking session. A hacking session is basically a test which exposes open SSH and waits until I say I'm done. But it's nice because from the outside you can go ahead and connect to the board. Okay, now let's see if that works. Let me take the mouse that's faster. That's lava, where is the thing? Okay, let's see here. So that's our Fuego. I'm using already the next branch. So that's not the master that's already next. Well, we have here such test plans. I use the Raspberry Pi as my device under test. And ignore the docker. That's just the test plan name. Anyway, so let's take a look at one of these. It's actually running right now. And let's take a quick look. So we start up and at some point over here we go into the link setup. We submit the job. Well, and then there is one thing now I don't know when actually SSH is up. So I do a poor man's, hey, SSH there. So it would be nice if I get the, for example, lava knows when, for example, the login was sent. I don't get that yet through lava tool would be cool. I think there is the, isn't there, it's not job, job details. Wait for there is some weight. But I think that's the plan to expose that, right? Okay, but I have to send the event in my, yeah. Yeah, well, here I'm in the, I'm basically in Jenkins, which has to do the flow control. Well, right now I just do net cat port 22 a couple of times and whatever. Yeah, it's a little bit poor man's polling, but it does the job. Yeah, and if that succeeds fine board is up. And then Fuego takes over over SSH. It does then it's job calls SSH a couple of times runs the job and so on. And in the end, yeah, I basically bring down the job. Yeah, at the moment I cancel it's a little harsh. So, and that is kind of a first poor man's implementation how to connect these two. The plus side, I get remote board support. I get multiple boards, multiple boards of the same type. Fuego still couples the board with one executor. So it actually just knows I can execute one thing at a time. If I now have multiple boards within lava, I can so I can say I have four executors and execute four of those test instances at the same time. Yeah, which is nice. Yeah, to scale that because they all run for I don't know how many minutes. So that's a plus and that also means it makes sense to attach that whole flow control to the job. Because then you can say, okay, multiple executors, multiple boards in the let's let's say lava is the orchestrator in that in that in that sense. And I can scale and gain speed here. Okay, so let's see. How to so it's yeah. Yeah, I have the link. Okay, so it's on bit bucket. Downside of lava, they. Downside of Fuego. They have they are it's quite large because they ship the source tables for the test in one thing in their git repo anyway. So basically you create the container. So there is Fuego and Fuego dash core core is the actual engine. And that is just a container. So basically here you create the container. And so on you create a board and note and Jenkins. This create this script basically creates the various jobs plus the wrapper, which calls it plus the batch job that calls it. I'm sitting on next already. So the newest stuff. It works meanwhile quite well. And I can run jobs quite reliably. Even if I get other jobs in between the question is it would be better to not rely on the calling and the time out. So that's something to to really improve the handling of the boards is then very solid. And basically with two scripts added. We are done the downsides. Let's see. So we had predefined timeouts in Fuego. A test runs three minutes. Yeah, wait a minute. Yeah, if I have to set submit the job and then wait for the board to become free. Yeah. So those hard coded timeouts were just bogus. It was at the wrong place because it was at the overall job. So we need to make those more fine grained and I'm discussing with the Fuego guys how to how to do that. Another downside here is the job in Fuego starts with build the test while it blocks the board in this setup. Yeah, so we grab the board just to start compiling, copy over the stuff, run it, retrieve the things. And then we also do the reporting before we close the board. Okay, that is not optimal. And we are looking high. Okay. What is the mineral set? Yeah, the copy over run retrieve. And then we want to small down the time we use the boards actually. Yeah, what would be nice if we get some progress messages that would allow me to skip the SSH polling and say, okay, the login was sent. The board is up. Yeah. That would simplify it to just a call to lava tool and done. That might be nice. Yeah, I have to figure out the way how to how to overcome the SSH for Fuego. Not sure what to do right now yet what pipe that should be. Yeah. So what needs to be done? Yeah. I think on top of next, that branch needs to be cleaned up themselves. So there's some work to do. I hope we have some something for like in two, three weeks or for ELC. Let's see until then. It might be interesting to still match up with the transport in Fuego SSH serial. Also, as you have seen here, I do one job, bring it up, tear it down. It might be nice to say, okay, pool five jobs, which means bring it up, do five stuff and then tear it down so some other job can take over. So that pooling might be nice. And yeah, for my use case, I need an alternative for the local SSH thing. So we can get the full remote lab workflow done and try it. The individual projects, Lava, Fuego, or contribute to projects like Kernel CI, highly recommended, I run a lab myself. What is important now with all those test frameworks, we should make sure that we use the same way to describe the test results so we can actually share those. So again, at that point, we should not reinvent the wheel and do the same test three times just because it's a different format. And with some shared test results, we can do better, well, analytics in the end. Because we can only learn if we see what changes over time. Okay, questions? Yeah, I did that. I have a timeout and I check if port 22 comes up or not. It works, yes. But with the idea of the satellite workers, I don't want a VPN. I just want some way to, Fuego just needs two things. It needs the terminal and it needs to weigh how to get the test files over. And if I get that, if I get a different transport, basically, I'm happy. Yeah, but that's something to spend some cycles on because, yeah, it needs to be secure. Thank you.