 Felly, wrth gwrs, yma'r hyn yw'r cyfnod i'w credu'r arferon a i'n rhan o'r problemau ar y cyfnod ar ôl? Gydag rydyn ni'n gyffredinol ar y cyfnod ar y cyfnod i'r cyfnod i'r ffordd. A'r hynny wedi'r gweithio'n gweithio'n cyffredinol i'ch creu'r ymddangos i'r awm, ac mae'r gwneud i'r gweithio'n gweithio i'r gweithio'r awm. Gweithio'n gweithio'n gweithio'n gyffredinol i'r gweithio'r awm. Ac mae hyn yn ymweld. Y rhan o'r myfyddiadau ymlaen, mae'n oed yn cyffredig o'r rhan o'r prydyniadau fel y pethau o'r rhan o'r rhan o'r rhan o'r rhan o'r broswn y dynodau. Mae'n dwylo'r rhan o'r prydyniadau a'r prydyniadau a'r prydyniadau, mae'n meddwl i'r prydyniadau ac mae'n myfyddiadau arwad. Mae gydag i mi i ddweud yn llunio ydw i unionid, ac mae'r prydyniadau hynny o'r prydyniadau. Ysdyn ni'n gweld eich eich cyfnod a'i gyrwch am bach yng Nghaerdydd yma. Gwyd ddw i ni nhw i ni fyddan nhw i gyd yr rŵn gyfytu. Yna wnaeth y Gadael i'r Pdf yn ei gweddyn nhw i gion i'r ffianfa yng Nghaerdydd nawr. Felly, i fe yw meddwl gyrwch am gylwyd y cwrdd i gael gwaith'r hwnnw i'r pethau. Csiwg gwybod gwirionedd wedi i chi wneud ar y wedi'u bod gennym nhw. summeryn yw'r gweithio gwirionedd y gwaith bwrdd i'r ffat, a'r hyn o'r rhaid i'r hwnnw gwaith hon oed i'r lab yma, felly mae'n cwrw'r gweithio'r hwnnw i'r hwnnw. I'w'r hynod i'r hwnnw i'r gweld o'r stag yw'r cyflodyniad hefyd a'r hefyd o'r pwysig sy'n gweithio'r holl yma o'r bethau i'r dwylo. Mae'r ddweud o'r ddweud o'r ffwysig yw. Felly go a gweithio'r ideaethau sy'n gweithio a ddwy'n gwybod i'r ffordd o'r ffordd o'r cyflawn. A ddyn nhw'n gwybod i'r cwestiynau o'r ddweud. Felly, rydyn ni'n CRIS. Rydyn ni'n gwybod i'r ddweud o'r ddweud o'r cyflawn o'r Llinux, IOT, o'r metalu, o'r ddweud o'r arsos, o'r ddweud. Rydyn ni'n gwybod i'r ddweud o'r ddweud o'r cyflawn o'r ddweud o'r ddweud o'r ddweud o'r ddweud. Felly, rydyn ni'n gwybod i'r ddweud o'r cyflawn i'r ddweud o'r ddweud o'r ddweud. Rydyn ni'n gwybod i'r ddebu'r cyflawn o'r cwestiynau wedi cael ei bod yn meddwl o'r cyflawn ymgymhau, i'r ddiwrnodi ar y Gymraeg, o'r cyflawn gweithin nhw'n ddweudio o'r cyflawn. Dwi'r symud yn ffoedd o gwnogell yr oedd, yn cyddol ymgylch, o'r ddangos o'r dail. I'm always keen to learn more and check out new things that are in the embedded world. I've always really been interested in board farms. Creating sort of set ups that make it easier for me to actually do my daily work. So, these days I'm remote working, and a lot of my colleagues are also remote working. It's really hard for us just to get a board to try a patch or something. ac mae'n meddwl o ran o'n meddwl ar y bwysig, ond mae'n meddwl o'r bwysig i'r ei wneud i ffnogi yma. Nawr, mae'n meddwl o'r bwysig ei'r bwysig yn y bwysig, mwy fawr ar y bwysig oherwydd mae'n meddwl yn yr un, rydw i'r hoffae sydd yn bwysig o argyrchu ffordd bwysig, mae'm rydw i'n meddwl o'r bwysig ond mae'n meddwl o'r bwysig i'r bwysig, Mae'r hyn yn gweithio i gynnwys i digad y wahanol i gynnydd f tertai ysgrifiadol. Mae wedyn gyflawni. Mae'r Sd cardiau fy oes bryd o'r rhaglen. Yn oes, mae'r sgwlad o'r fawr fod ymlaen. Mae'r Sd Cardiau fy oes bryd a'r hyn o'r hyn o'i gwbl fel yma'n gwybodaeth gydig. A wedyn y gwaith y mae'n gwybodaeth. Dylai'r dweud o'i ffordd oed yn ymdwy. Dylai'r dweud o'r mae byddaeth yn gwybod o'i ffordd o'u ddeg. ydych yn y r Garyc arloed yng Nghymru. Ond mi wnaeth chi'n gwybodaeth sydd yn Unedig Brasil a'r Indie. Yn cael rhai llwyddi angen i'r bobl. Yn ceisio bod y gallwch hynny achos chi'n gyd yn y bobl. Rwy'n dweud, mae hwnna'r aplyniad yn syniad rydw i'r cystafoli yn dweud roedd yn gweithlo. Rydym yn hynny byddai'r gwmorth o'r hardware rydw i. Rydym yn golygu o'r bod arall yn ddechrau o ddaeth chi'n o'n gallu syniad ar y bobl yna'r byw'r bwysig o'ch cyfrifnig iawn. Mae'n gweithio, mae'n gweld hynny hefyd yn ei wneud ar y tessu automaticau ac mae'n ddiwedd y bwysig hyn o ddegwydau gidiau a'r bwysig y tylu mewn eithaf hyn. Mae'n gweithio'n gennym o baitfawr i hi'n gweithio'r hir y tessu automaticau In May this year, I had a goal to set up my own home lab. I really wanted to start small simple and for it to be really cheap as well. I didn't want to spend loads of money on expensive test equipment or anything like that. But goal was, to have around 10 devices in the lab as a maximum a make it easily accessible to anyone else to be able to create their own home lab without having the same things. I want to be able to add, remove and reconfigure devices really quickly, in some cases it can take a long time to set up devices for being accessed remotely. I wanted it to be really low power as well so I didn't want some x86 server or something like that running all the time, taking up all the power. Also it needs to be quiet and take up a small amount of space basically for a good wife acceptance factor. She comes in my office and says it's messy in here, you need to tidy up so I don't want any more problems there. Also I didn't want to cut any wires, do any soldering or have anything like electrocution risks because not everyone's able to do these kinds of things. So it wants to be really simple for anyone to get started. One of the main goals was just to be able to share boards with colleagues. I mean I've got lots of boards, you know, every conference I go to I get another board, order all the time from AliExpress and I want to be able to share them with colleagues and let them actually write their own patches and things for those boards and be able to test it without having to ship the board to them back. It's just too complicated. Also we wanted it to work with non-linux boards so there's a number of projects these days that are running embedded RTOSs like the Zephyr. It would be really nice if we could have our board farm have those kinds of boards in. And we also wanted to run automated tests on it. We have like nightly GitLab CI builds of images and kernels. We'd like to run that on hardware and report the result almost instantaneously. And amongst all of this, the most important goal was to be able to help others set up their own lab because it's kind of a bit of a black art. It's quite hard to get it set up so the main goal was to make it easy. And this is what I came up with. So you can see I've got four embedded boards and a Raspberry Pi. Basically the four boards, the device is under test and the Raspberry Pi is sort of the controller. Obviously good luck if you want to get a Raspberry Pi these days. So maybe it isn't such a simple setup after all. There's also on there a network switch which are connected to all the devices. And all the devices have console over USB serial ports. And they're powered using the sonoff devices at the top which are Wi-Fi relays. So most important thing for me as I said earlier was to not be able to not have to cut any cords. I wanted to make it simple and easy to be able to swap out a power supply and swap out a board. So it's incredibly simple with this setup to take a board out of the system and put another one in. If something goes wrong you can easily remove it and put something else in because it's quite modular. So here I've only got four boards but there is additional space here for me to have more if needed. So I wanted to talk about the existing solution that we use at Collabra for automated testing lava. It's by Linaro and it consists of a web interface and backend in Python. And it's really a scheduler for tests. It schedules the tests in like a batch mode and then runs them on the hardware, reports the results and keeps the logs. Now because it's a batch system you don't really have the ability to interactively play with the board. And these tests they're written in YAML and the execution steps are defined in the YAML file. These YAML files can be in the hundreds of lines really. So it can be quite difficult to be able to use lava in this way. Also it needs a database to keep the state and the logs which in some cases the database can get quite big and probably not really suitable to be run on a Raspberry Pi or something like that at home. So KernelCI uses lava. It's got a high buy-in from industry and there's heavy maintenance investment involved in it. There's other projects that use lava, so MSCI. And also there's other things in the V4L2 area of the kernel where we're trying to add in support there to do integration testing. We at Collabra have a lab in Cambridge with 45 different devices, which has got a total of about 450 devices spread across those different device types. It's always growing as well. I mean we're adding boards in batches of 10 at a time. And anyone at Collabra can submit these test jobs and they can run on real hardware. If you want to hear about the struggles that the lava lab has had recently and how we've got around them, Laura, my colleague, has done a presentation yesterday on that. It's available to view online. And she's also somewhere, if you want to ask her after. Is tomorrow, is it? OK. Is tomorrow. I thought it was yesterday for some reason. So here's a snippet of half of the lava farm. It basically is full of kind of server racks, 90-inch server racks, and each place has got a device inside it. Some slots have got multiple devices in. Essentially, I wouldn't get away with having something like that at home. My house is too small and I'd probably end up with a divorce. So it's best to leave that where it is. I don't think that lava is suitable for a home lab, mainly because it's suitable for testing specific parts. So if you have a kernel patch or some bit of user space you want to test, that's where it is really suitable. It's not suitable in my case where I want to be able to access a board remotely and actually run interactive tests on it. It's also only suited really for Linux systems so you can't really do any testing with embedded boards like things that are running RTOSs. There are some cases with lava that it's actually really difficult to do basic things, like just flashing a full disk image to the SD card or the MMC. You've got to do some work arounds to do that. Mainly the feedback loop for debugging test is much too long because you have to submit your job and then wait for the results. There could be other people who've got more important jobs and it takes a long time for the test to run so it's just too long in my opinion. As I've said, it's hard to connect to the board interactively. There is some work there with hacking sessions in lava and there's also an old project lava bow which allows you to do that. From what I can see, it's not the main goal of lava to have interactive sessions. My personal opinion is that it's too complicated to set up and maintain for a home lab. The last thing I want to be doing is maintaining my home lab. I want to be developing things and making sure that the time is used effectively for working on actual patches, things like that. I'm not going to go into any of the other solutions. There's a page by Tim on boardfarms on eglynx.org. The long and short of it really is none of them, in my opinion, are suitable for interactive usage. You may be able to prove me wrong there. I think you might. I keep talking about interactive usage but what do I mean by that? I mean actually being able to access the board and control it remotely so as if you've got remote hands there. We want to be able to control the power, turn it on and off and see whether the board's turned on or not. Maybe even do things like get the power levels of the board, how much current it's using, that kind of thing can be useful. We want to be able to flash firmware and we want to be able to boot into the software. So you may want to boot using something like TFTP or you might even want to flash the whole EMMC or SD card and boot that way. So when we've got our software booted we want to be able to get into a shell and really we want to abstract all of that away from the user. So what I mean by that is we don't want to be running commands like getting a shell up using mini-com or something like that because really you can do that. You can just connect your board up to a machine and run a mini-com or something like that to get a shell but we don't want to remember all the settings. We don't want to do anything like that so we want to abstract all of that away and make it nice and easy for us to connect to the board. We also want to know who's using what and be able to reserve hardware so you don't want someone interrupting your hardware testing. You don't want the CI system to come in and push a job and you don't want it to just interrupt your session really. So LabGrid is a tool by Pengatronics. There's some people from Pengatronics here today who hopefully will be able to answer any specific questions later on. It's a CLI tool so you can interactively communicate with your boards. There's also a Python library which allows you to run automated tests or more complicated tests. It's really usable for daily interactive development as well as debugging I think. It's really simple just to be able to get started and it's really good because it abstracts the hardware into functional topics so you can ask LabGrid to turn a board on you can ask it to get a console you can do things like upload a file to the board that kind of thing. So it's written in a way that the topics that you're going to be doing actually make sense so it would be how I would tell someone to talk to it would be how I would tell someone to actually interact with the board like turn it on, flash the SD card those kinds of topics rather than any complicated words or complicated uses. So there's three bits of software in LabGrid there's a controller which handles the overall system state you can connect many exporters to the controller and the exporters run close to the hardware on a machine and it controls the specific hardware things like turning the device on connecting to the shell and then to the controller you can connect multiple clients so for instance your PC or your CI runner connects to the controller to then talk to the exporters which then talk to the devices. The controller and the exporter you can put them on the same machine that's what I've done because my lab's quite simple a controller like I say has multiple exporters that connect to it and the exporter has places and a place is a collection of multiple resources and a resource is just a driver which connects to that specific board so some examples of resources you've got a network power port which is a power port which implements functions that a power port could have like switch it on, switch it off, get the state there's other resources like a USB serial port again is a serial port which implements a function that lets you get to a console so it's basically using interfaces and abstractions to make it work with any hardware so one real bonus about lab grid is it's so simple to configure so here is one of the places that I've configured in my lab and it's basically the devices which are on the exporter and there's a tool called lab grid suggest which you can use which when you plug in the device it actually helps you build your exporter YAML file so here you can see that I've got one device that has got one place that's got two resources as part of it it's got a power port and it's got a USB serial port and it matches the USB serial port based on Udev paths or Udev properties and it's got some other properties there as well so like the speed that you want to connect to the serial port under the power port that I've connected is connected over MQTT and I've defined here the topics as well that are used to turn the thing on turn the thing off and see what the state of the power port is as well so then there's also a places.yaml configuration which is for the coordinator and this is generated on the client side by the commands that I've put here so you can basically set the tags up and you can add the match to the place as well and it's stored as YAML because it's then easy to store under git and then you can diff things and basically see what's changed so the YAML configuration is really simple compared to a system like Lava where the configuration is not so simple in my opinion and that is basically it the configuration that's needed to actually set your farm up it's to YAML files so then if you want to use lab grid to connect to your board and you've already set your lab up it's real simple you can just install lab grid from your distro's package manager you set the crossbar URL which is basically the path to the the exporter that you want to talk to or the coordinator and then as soon as you've done that you can list the places or you can just run lab client to do other communication with the farm so here I'm listing the four places that are in my farm and each place relates to a specific board in my farm so then if you list the resources you can see the resources for each of those places and it consists of the host name of the exporter it consists of the actual place name as well as the name of the driver that's used if you add a verbose parameter to this it spits out so much information as well so you can see what is available to you what can be used so lab grid has this feature where you lock a board for your, well you lock a place you require a place for your usage and then when you're finished with it you release the board and let other people use the board there is also a way that you can wait for a board to be released actually then lock it as part of a CI job but I won't go into that detail here I'm just going to go into the interactive use case so the steps really are you acquire the board then you can do what you like with it and then when you're finished you can release it so if you've already locked the board here I'm trying to acquire a board that's being locked by someone else you can kick people who have actually kept the board if someone's locked a board for too long then you can kick them off and then acquire it yourself one thing that would be nice is some sort of time out feature to automatically kick someone after they've had the board for a day but so then after you've acquired the board you can do things like turn the power on by just saying lab grid client power on lab grid client power off or you can cycle the power as well so they're real simple sort of verbs that tell the board tell the farm what you want to do to the board you want to power it on, you want to power it off and then to get a console on the board you just use the verb console and then you get a serial port it's that easy really so I mean I've only gone into a few of the drivers that are available in lab grid but there are plenty so you can get to a console with SSH serial ports USB serials or a network connection telnet that kind of thing to communicate with power ports you can use PDU daemon which supports hundreds and hundreds of different types of PDUs you can communicate with POE ports turn them on and off you can use the open test motor firmware which basically is for some ESP devices and then you can create your own boards that have got ESP devices on and be able to turn your own relays on that kind of thing it communicates with USB relays you can communicate with GPOs to turn on relays and also the really nice thing is it communicates with sigrock which is a library around bench power supplies and other bench tools like that so you can basically power on and off all of these different libraries just with the one power driver you can flash to IMX Rockchips you can flash using fast boot and there's plenty of other things that you can flash to like you can flash to an SD card using an SDmux or a USB drive you can communicate with digital IO as well so you can communicate with GPOs and relays that do other things not just power maybe you want to toggle GPOs to set the board into different states you can do that also there are other drivers for things like communicating with webcams capture, audio, HDMI so basically there are hundreds of drivers and well maybe not hundreds tens of drivers and they're just implemented as a simple python class that wraps the device you want to communicate with so in my lab I wanted to flash the boards but I didn't want to use SDmuxes because SDmuxes I don't know if you know them but you basically plug the SD card in and then the SDmux will either allow you to flash the SD from your PC or it will communicate or it will allow the SD card to communicate with the board under test they're quite expensive and I didn't really want that want to have that overhead in my lab you can flash over JTAG using things like OpenOCD which is useful for some of the boards like the Arthos boards that I was talking about earlier you can upload using fastboot Rockchip bootloaders like I say the IMX bootloaders in my lab currently the boards are set up to TFTP boot so they've got a bootloader on them that then just boots using TFTP I manually copy the files that are built into the NFS drive and I've built some images that just have bootloaders on you can click the link there to find those they're built with DeboS and they just set the boards to go into TFTP boot mode there is some discussion at the moment about how to get rid of the manual copy of the files because that's the one pain point that I've got at the moment in my setup so again back to the picture of my lab I've got a Raspberry Pi 4 that's running the lab grid co-ordinator and exporter I've got a USB hub that's got the UART devices on I'm having a number of problems with the USB hub at the moment for that the devices are on a VLAN basically with the they're running through a USB 3 USB 3 ethernet card which basically keeps the devices on their own network that's away from the away from the public facing internet and the power delivery as I say is controlled by the 4 sonoff sockets at the top so the software on the Raspberry Pi is basically running USB all I've had to do there is change the host name and run the Ansible Playbook and Ansible Playbook sets everything up on the host including all the various demons and everything is actually in a container the one thing that I haven't got in this setup is a way to add your own configuration but you can use my configuration as a sort of example if you like so again the playbooks are on github if you want to take a look so a lot of people will use a relay board in their lab to control the power on and off I decided very early on that this was not good because it needs the cables to be cut and then you can't use the cables or the power supplies for other things so it's kind of like a one time use really can cause the wiring to be messy I mean it's quite a difficult topic because I think that the wiring in my lab is quite messy anyway but it would be messier I think if there was a relay board in the way and the main thing really with relays that make them unsuitable in my opinion is they're supposed to be switching AC when you switch DC high currents with them it can reduce the lifespan of the the relays the welded contacts that kind of thing so I just think that using the relays just really isn't suitable so I ended up with these sonoff wifi sockets and basically it plugs into the mains of your house and then the other end you plug into the adapter that you've already bought so you can plug any adapter in it's really quite simple you don't need to do any soldering cutting of wires there's also a little button on the side of the thing so you can turn it on and off manually one of the things with these relay boards is you've got no way of turning it on and off manually without software in the way so if you just want to turn a board on or the software's not running properly as it should then you know you're going to have lots of problems and pain trying to turn the thing on and off they're fairly cheap I mean I think I'd pick mine up for about six or seven US dollars each in a pack of like 10 it's incredible they seem to be quite reliable I mean as with anything from the east you've got to make sure that it's been built correctly so I mean I opened all of them up and made sure to check that there was no leftover flux on the board or anything like that I've written some firmware with ESP home for these boards basically they come up as MQTT they come up as MQTT so you can easily turn them on and off using MQTT and also you can update the firmware quite easily over wifi with them it supports over the air upgrade so the ESP home firmware that I've written is available for those also a side note I've got home assistant running at home and you can turn the relays on and off using that quite easily and also there's a PDU at the bottom that is another sonoff device which actually supports power measurement that I'm trying to implement at the moment so I can actually get a feel of how much power the whole setup's using and it's also a kill switch because everything is hidden behind the behind that one PDU so you can easily turn off the whole setup when I go away hopefully there's no fires or anything like that in terms of the console I'm just using standard USB to UART adapters cheap clone FTDI adapters as I say they can become buggy with the USB hub so it needs a little bit more investigation from my side in some cases the bugs are just it stops communicating it can happen after an hour or after two weeks it just needs a little bit of investigation to find out what it actually is I'm also investigating using silicon labs adapters which I think CP 2102 or something like that and they've got additional GPIO pins so rather than having a separate relay board I can use the GPIO pins that are on the USB to UART adapters to do various things so if anyone wants to use my I don't know who that was anyway if anyone wants to use my lab they can quite easily jump in using an SSH tunnel lab replyant supports that out of the box the only thing here is the exporters and the devices the exporters need to reach the devices locally and the exporter needs to reach the co-ordinator but other than that as long as you have a SSH tunnel to the co-ordinator it's really easy I mean I go personally through a VPN back to home but this is the perfect way to let others use the lab so in terms of using CI I've got a GitLab runner that runs on the same machine as the exporter and a small shell script that basically just mirrors what you would do in interactive mode so powering on the device locking the device copying the files running the test scripts and the command on the board and then powers off the device when it's finished and reports back to state you can also write your own tests for lab grid with pie test I've not done anything with that so I can't really comment on it but there are plenty of examples on GitHub so really I wanted to talk about the future improvements that I would like to make to lab grid all the things that I see as pain points so as I said before I'm running Docker containers in my setup the Docker containers that are available at the moment are just for AMD64 so I've got a branch that I'm working on at the moment that builds the containers for ARM64 that then run on the Raspberry Pi I'd like to add in ESP Home API support at the moment I use MQTT but it's just a big kind of mess in my setup at the moment I think GPO drivers for the USB serial port I talked about I've got lots of all-winner boards and I'd like to be able to flash those using the lab grid setup so that should be quite a simple driver to add and also I'd like to be able to upload files to the exporter using lab grid client as far as I can tell there's no ability to do that at the moment so I have to manually upload the files that's just a small pain point for me some improvements that I'd like to make to my lab I'd like to document it, I'd like to tidy things up a little bit I'd like to add some more boards like I say I've probably got about 30 different boards that I'd like to add I mean it would be really nice to get them actually all running the biggest pain point in the setup is the PDU slots maybe I'd like to create an open hardware PDU if anyone's interested in talking about that after please see me it's the biggest problem in the setup because if we go back and take a look at my my setup it's basically taken up half of the space and it's just a bit of a kind of mess so that's one thing I'd really like to solve so if you want to learn more about lab grid I've put some links here basically there's a setup video that explains exactly how to set the thing up I think it's about half now it's a really good little video I've linked to the documentation and the source code as well I've got some special thanks I'd like to give some companies that donated the boards that I've used in my lab and I'd finally like to thank Jan as well for reviewing my slides at such short notice it's really appreciated and with that I'd like to have any questions you can come later yes please exactly I think really it doesn't matter because you can have many different devices but it's just how many run concurrently with a console so I'm looking really to add in more USB slots more console slots in future the original goal and motivation behind this was to actually do some research to see if that would be the case so I wanted to set up a homelab first and then based on that figure out whether it would be useful to set up a larger lab somewhere for use of interactive mode so I think that's the next stage is to set up a bigger farm and see whether it scales exactly yes I mean you can connect up GPOs to the system and you can put it into a flashing mode and then yeah so I think that comes under a little bit more of the the testing modes of it so you can set the board up into different modes in like a flashing mode right so I think that the reason behind calling it a place please correct me if I'm wrong is that you don't specifically have to have a board in that place you could have a place that allows you to put in different boards into that place so you know for systems where you can remove the board and put in a different one for instance I mean that I would personally say board and place could be used interchangeably but it's a design decision that maybe we'll discuss later the back please I've not looked at that personally but there are options in Labrude for that and there's plenty of demos for that on the Pengatronics YouTube site so again I've not looked at that personally but I'm sure that there is options for that yes please so at the moment just to make it really simple I've just got one VLAN for all the devices so all the devices could talk to each other but that would be a good next step VLANs per device I'm not sure on that the answer's no so I found in my own lab that I changed the wires around enough that maintaining the mapping between like the USB ports and the ethernet ports it's just it always gets out of date I don't know if labels for me labels would be hard to make so thank you very much for coming and thank you very much for listening so patiently thanks