 OK, so, good afternoon, and welcome to this bird of a feather session on farming together. It's actually a bird of a feather session, but it doesn't appear to be on the agenda as that, so apologies if you've come expecting not to participate. If you're not familiar with this format, a boff or a bird of a feather session is one where the presentation is slightly lightweight, it's less formal with a view for us to discuss and collaborate and find out how we can work together. So feel free to pitch in at any point and raise your hand or say what you need to say. I'm going to divide this into two parts. The first part will be me talking about my view on the ecosystem for farms and the second part is us collaborating and discussing what we need to do to improve. Before we begin, perhaps we can have a show of hands. Raise your hand if you currently have a farm. You're considering making one or you're maintaining one. OK, nearly a full house. I guess that's why we're here. So I'll share with you my idea on what I think a farm is. In my view, a farm is some hardware, a system that allows you to interact and automate interaction with embedded devices, such as reference boards. The purpose to that is so that you can, in an automated way, turn on a board, put some software on it and run some tests. The benefit of this, of course, is that in your continuous integration flow, you get to add two new boxes. After you've used something like Jenkins to build some artefacts, you can automatically deploy them and put them on your board and then you can automatically test them and then feed the results back into whatever comes next, usually perhaps some email notification or similar. The picture in the background is also what I consider to be my ideal farm. Something that looks beautiful, lots of rack mounts, organised, beautiful cables, structured, all these lovely words. But I think if you've already got a farm you'll know that it's really hard to do that and that's not quite the reality. So, this is the reality quite often. This isn't my farm. These are some farms I found online. They might be yours. Does anyone want to own up? Anyone recognise this? Are they the same? Oh, okay, sure. I think, yeah, three electrons. I'm not sure about the other one. And it's because it's difficult. It's really difficult to manage a bunch of hardware and cable them all up to the same place and pair of supplies. And they're very diverse. Every person's farm will look quite different. Some use shelves, some use drawers, some you just use the floor. And in fact, a quote I found by someone describing their farm is, it's a pile of stuff, which is accurate. So, a picture of our farm. Ours is still a sprawling mess, but we did try to put it in a rack mount. We had some good intentions. Originally our farm was designed when we had no office so that our home workers, our employees could access customer hardware and share hardware. And this saved shipping hardware back and forth. It meant we could protect our customers' hardware and do all the right things. As you can see, we have a rack mount. We have shelves and hardware is on the shelves. And all those reference boards are connected via USB and ethernet to one machine, which we call the farm machine, which is just a desktop PC running the next distribution. If you want to use the farm, you SSH into it, and then using some of our proprietary scripts, which we call EB Farm scripts, you can do things like turn on a board, look at the serial console, whatever else it does. Our board farm is part of the Kernel CI project, which we can talk about in a short while. This is a project where the community develops new kernel releases. They're automatically built and deployed on farms across the world and some of our boards are part of that. So, on a daily basis, our boards are being used to test Kernel's boot. We also use it to run automated tests for some of our customers. The capabilities of our farm, we can control the power to boards. We can turn them on and off using USB relays. We can press buttons or flick switches. We designed our own SD mux hardware. This is something that allows you to change the contents of an SD card, so that if you've got some hardware that boots from an SD card, you can completely control what boots. At one point, we had a HDMI receiver card, so that if a board has HDMI outputs, it was possible to Skype into our board farm and see what's going on, which was quite useful for the project we had at the time. We've also got some concepts like workspaces, so if you want to use a board, you can create a workspace and have TFTP NFS in a convenient way. We do all this through containers as well, so that when you ascertain to the farm, you have your own workspace, so that you're not going to break someone else's environment. Slightly closer look. This is a farm shelf. What we try to do is put each unit of hardware on a board, like a plastic sheet, and we secure the hardware to that board with things like velcro or tape or pillars. We put everything we need for that board. You can see that we try to have some common interfaces, so we have for each board power, ethernet and USB. We try and make that uniform across the farm. At one point, we thought about having a USB stick that's on each shelf, which describes the topology of the USB devices in it. The idea being that if you have a shelf, you can plug it into the farm, and the farm automatically knows what it is, how to access the devices on it, so you don't need to do any kind of configuration. Of course, the idea being that you can remove a shelf, work it with it on your desk, put it back in your farm, perhaps someone else's. In this particular case, this customer, we help them with surveillance software, and their update mechanism uses SD cards. In order to test that, we use our farm, and we test that we can upgrade to different versions of the software. We can do that hundreds of times a night, which saves a lot of manual intervention that would otherwise be needed to do that. We also run CPU load tests over about an hour so that we can get a good profile of the latest software to see if there's any regressions in performance and some other metrics. At the end of this, we get some automated emails and test results that our customer finds extremely useful. This hasn't been easy. If you've played with farms, you'll be familiar with some of these issues. I'm sure you have many others. It seems like farms are quite a good test for USB. I don't know if I just buy really crap hardware, but quite often hubs disappear, USB to serials disappear, and of course they come back with slightly different major mining numbers, which confuse lots of things, especially if you're running a long test. Reliability is the biggest challenge. It's easy to make a test. It's easy to do this stuff, but then to keep doing that consistently and reliably is difficult, especially as you try and scale the farm, which is a challenge. SDMux, it works for our needs, but it's not reliable and it's been very challenging to get that to be useful. For our containers, we used OpenVSed. I think in hindsight that was a bad choice, and that's created lots of obstacles for us. Simply maintaining the farm, keeping it from sprawling, keeping it tidy, stopping people from poaching hardware from it and cables when they need it, is all quite difficult, and we're managing this through our own time rather than some IT department, and of course if something breaks, you need someone to spend time to investigate, and that type of support infrastructure is a challenge, especially if sometimes you don't know that a board is broken in some way in your farm. We do a lot of testing with TICL and XPECT, so some of our tests, they boot a board, wait for a prompt, and then execute some commands on there. But I don't know if this is testing in general or TICL, but I find that you write a test, you run it, it doesn't pass, and you run the same steps by hand manually, and it does pass, and there's this really difficult, intermittent type of issue, and it's really hard to make tests that keep passing, and you find quite often it's because of infrastructure issues. But I think if I learn anything, farms seem to be the latest trend. I think a few years ago it used to be boot time, but now people seem to move on from that and looking at things like farms, and I think there's a reason for that, and I think it's because farms, whilst they can be quite cool in themselves, they're quite enabling, they allow you to do automated testing, and that's what truly provides value for open source development, but also commercial development, because they provide a mechanism to maintain a level of quality and to spot regressions. It's also having quite a big impact on the open source community. I think the most notable one is KernelCI. I briefly mentioned this earlier, but KernelCI is a project where every time, a number of trees are monitored in the kernel community, and when changes are made to those kernel trees, servers around the world build, those kernels for lots of different configurations, and using farms like ours, and I think there's another perhaps 12, those kernels are deployed and booted, and if boards don't boot, that feedback gets sent back to the community, and regressions can be fixed, and even something simple like a boot test is incredibly valuable for the kernel community, because when you support so many different boards and so many different configs, it's really hard just to make sure they all still boot. So that's having an impact. There's EOS ADL, I believe they have a lab, and they use that for ensuring that the real-time patch set doesn't regress. I think they probably run a psychic test to check for a real-time latency. Qualcomm learned to have a board farm to test their open WRT distribution. Don't know much more about that. Intel have a zero-day test spot. I think it's predominantly x86, but nevertheless it's another example of where we're using automated testing to improve quality, and of course there must be huge deployments in private companies to test products un-similar. But in my view, I think we're touching the surface. I think we're all individually making farms, but we're still at the basics. We're still at the can you boot a board? Can you just test connectivity? We're not going to be on that. I think we could do more with boot time, with power management. We could even add interfaces to test that's outputs from the board like audio are as expected. I think there's so much more we can do. But the problem I see is that there's perhaps too much diversity. It reminds me of, I guess in the history of computing, when computers first came about, every manufacturer would have their own architecture, and there's lots of interest in developments, but in silos. Everyone's working on the same problems, coming up with different solutions, and there's no interoperability between them. That's the same with farming. There's no blueprint for how you make a farm. If I asked you how do I make a farm, I would get different answers from everyone. There's no genuinely accepted way of doing this. There's no resource available that's like a wiki, for example, and there's very little collaboration in terms of software that can run. Tim, there's a farm main list. Exactly. So it's not a surprise that everyone comes up with their own solution. And I think in the presentation I was at last year, someone talked about using Lego to create a farm. Is that person in the room? Yes. So it's your rack. I guess it works well. But I think generally we're all facing the same challenges. We're trying to make something scalable, something reliable, something useful that we can use as a platform for testing remote access to boards. I see it this way. I see there being three different aspects to farms. The bit that gets the most attention is at the top, which is the kind of continuous integration type of DevOps type of stuff. Jenkins, most people are familiar with. I mentioned KernelCI, which sits on top of Lava, which is provided by Linaro, and what other proprietary test frames you might have. I think that part of the system is quite mature. I think there's lots of support, lots of documentation. The bottom part is hardware. This is the physical hardware. If you've got PDUs or some kind of network controlled power supply, USB relays. We talked about SD-muxes. I think there's a lack of information in this area where collaboration would be useful. Simply knowledge on what USB devices are good quality, which ones aren't, with the SD-mux. It turns out lots of people are doing this. I know that Linaro have done this once. We've done this. I think Tizen have open schematics for their SD-mux. An example of at least three different companies doing the same thing. Maybe with better collaboration, there would be more awareness of that and maybe improved devices. Bailibra, they have probes for measuring power consumption. I think there's some activity in this space, but quite often not everybody's aware of that. The bit in the middle, the hardware abstraction, I think this is where the biggest gap is. This is where you bridge the hardware for example, if you've got a network power supply, someone still needs to write some code that knows where that is and what command to send to that to turn whatever port on and off. That needs to be linked with the top player like Lava where there's a hook that says, how do you turn a board on? This is a bit in between that I think is missing. I think this is the part where my suspicion is that everyone comes up with their own solution. I know that free electrons have something called Lava bow, which seems to extend Lava to allow you to remotely access boards for development. We have our own EB farm. Can you raise your hands if you've got your own? TTC, OK. What talks that? OK, thank you. My vision is that if we come together and I think there's 50 or so people in this room that have some interest in working together, I think there's a lot more we can achieve together. I think as a minimum, an achievement that I'd really like to see is that we know who we all are so that we have some connections that we can not reinvent the same wheel. I think that will allow us to move out of the basics, simply turning boards on and off and doing some farm work to some more really interesting stuff like testing USB devices by removing them and reinserting them or similar to them. OK, and I think Leonardo, I've seen, have some cube-looking device that can stack to do similar things. So I'd like to move on to the open discussion. I split this into two parts, but I'd like to know who everyone is. Obviously you don't have time to do everybody, but what we're doing with farms, what problem we're trying to solve, what challenges you face, what solutions already exist. We mentioned some other TTC and different technologies, simply knowing of their existence I think is really valuable. And how can we calibrate? Tim mentioned the lack of a mailing list, maybe that would be helpful or maybe a wiki. I'd then like to talk about what it is we could do. Is it as simple as just having a place for information or do we go beyond that and create this middle layer that can bridge a gap? Wouldn't it be nice if when you buy some hardware like a USB relay, there's already a kind of driver or a config for this middle layer so that when you want to set up a farm it's kind of a case of app getting stalled and it already supports whatever hardware you've got. So perhaps we could start off with introducing ourselves. Is there anyone that has a farm that would like to tell us a little bit about themselves? Okay. Can everyone hear Tim? Or do we need a microphone? Okay. So I've been doing this a long time. Actually I started a test lab in the early 2000s and that's why we wrote TTC. So I've been using TTC for board management. Doing Fuego is the test framework on top and a variety of solutions that have evolved over the years for the stuff in between. And I just want to say, but I only have like two to three boards, right? I don't have a lot of the kind of more complicated stuff, although I did just get myself a 28 port USB hub so that gives me room to expand. But what I want to say is if you don't introduce yourself here, I really think we should get on a mailing list or get some or a wiki at least with a page with all of our names and email addresses so we can communicate with each other and talk about some of these issues. If we only got that out of the session, I think it would be a great start. Okay, thank you. Yep, I've got a question now. Perhaps I'll come, I don't know if this works. I'll come down. So my name is Kevin Hillman. I came up a few times, I'll introduce myself since I'm kind of one of the founding members of the Colonel C.I. project. And I have one of the largest labs contributing to Colonel C.I. I have about 80 boards in my farm. And those lab, the pictures he showed there look super clean compared to my lab in my lab. I was going to say, that's not messy. Yeah, those are, to me that was like, man, that is clean, yeah. In fact, when most people come to my lab they usually ask me why do I collect cables? Not why I collect boards. So if you have questions about Colonel C.I. and things like that, I'm happy to answer questions as well. And one thing else I was going to mention in the SD-mux category, there's these Wi-Fi SD cards now that actually basically make SD-mux, or SD-mux is, you don't need them anymore, or they're a little more reliable. And I know some of the U-boot maintainers are using the Wi-Fi SD cards, or the Wi-Fi SD cards actually just to upgrade U-boot all the time over Wi-Fi. They seem to be just as... They're not 100% reliable either, but they're more reliable than the SD-muxes that I've seen. Yeah. The Wi-Fi SD cards. Yeah, they're SD, regular size SD cards that actually have Wi-Fi in them so you can actually change SD contents over Wi-Fi. Which is... We'll use something. Yeah, ship it. Thanks, Andrew. I think... A thought occurred to me when I was listening to this. Thanks, Andrew. Doesn't Libvert show us a way to solve this problem? I'm not saying Libvert the technology, but the guys behind Libvert came to the conclusion that they weren't going to create a single hypervisor API to control them all. But the idea to put a common wrapper library that allows people with bespoke back-ends to write a little shim that would drive their special exciting bit of hardware and people writing wrapper GUI, continuous integration, Jenkins plugins can target that bit of middleware. You were talking about that right before lunch? Yeah, but they were kind... I sat in that, and the guys were doing great stuff, but they were writing, this is how we solved our problem. So it was more a story of their project rather than any kind of idea to produce some sort of consistent model. What are the... I'm really interested. What are the verbs? What are the... What is the API between the upper layer and the middle layer? And Libvert has an API, and so maybe that's a starting point, but we need to... If we could define that, like you said, if we could define that, then you could... The board or the... What I'm now starting to call the DUT controller, the device under test controller, could come with a driver that just plugs into that and provides that API. That'd be super nice. Anyway, I don't want the mic. Okay. I don't really want to hijack this discussion, but basically these ideas and requirements were what drove us to start this lab group stuff, to have something between the physical hardware and the automation layer, like Jenkins and PyTest and so on, to abstract hardware control for testing for other stuff. And the codes on GitHub and Lab Grid. And we use it in our lab, which has like 120 boards from our customers for CI and also for interactive control from our colleagues who sit at home and access those boards. Thanks. It seems really interesting, I'll have a look at that later. Did you have a question? So I'm seeing a lot of, like the consulting companies using this technology and providing board farms. What can we do to get the original equipment manufacturers and the SOC vendors to actually do this and contribute their results back as well? Any responses on that? Fuego, which is the test framework I'm working on, has capability to send test results in to a common server. It's not as mature as the API that like kernel CI has. But I don't... And we have some large companies that are using it. But nothing's gone public yet, so... Okay, thank you. Is there anyone else that would like to... Yep. So my name is Heert Uthroven. I gave a presentation about creating a board last year. So my main motivation was to get the boards off my desk and into a farm. So right now I have a board form of 8 boards controlled by a Beaglebone Black. For power control I use the Bay Libra ACMA Cape for the Beaglebone Black. The Beaglebone Black also does serial consoles. And for pressing buttons and resets, resetting of the board I use a board with optocouplers. So last year had just something on a protoboard, but in the meantime I had a real PCB for it. I uploaded the keycap files to Github. And you can actually order the board from Eisler, which is some makers PCB support small company here in Europe. And that's it. If you have questions. I have one of the boards here if you want to take a look. It's my first PCB ever, so it's probably not that super fancy. Thank you. Thanks very much. Any other experiences anyone wants to share with their board farms? OK. So we're making some good progress. We've got an e-wiki going. We've replaced SDMux with some Wi-Fi SDMux. We've got some high level software to look at. Can someone suggest any problems they currently have? I've got some questions on things that could be done. It seems like there's some projects ongoing that I wasn't familiar with. Anything from anyone? So you had mentioned like USB devices disappearing and so on. So one thing I've noticed with 80 plus boards I've bought lots of USB serial adapters like probably many of you have tinkered with. And I've noticed that a lot of them just after some of them just disappear after a day. Some of them disappear after a couple months to unplug them and re-plug them and they just show up again. And so I've just noticed that the cheaper of these cables they have more and more of these problems and you end up having USB devices disappearing all the time. And so I do not work for FTDI so I don't want to... This is not an advertisement. This is experience. But I've found that FTDI cables actually are just... They just work. They're just reliable. They're a lot more expensive than a lot of the cheaper ones. But they just work so writing Udev rules and stuff for them to show up actually is quite reliable. So that's one little tip if you're doing this just spend a little bit extra money get your FTDI cables and then you don't have to go out and unplug a cheap cable and plug it back in on a Sunday afternoon when your farm stops working on. Yes, so that's the caveat. There are a few boards out there that embed FTDI controllers on the board and they do not put an EEPROM on them so they all show up as ID0 so they're still reliable but you still have to write an FTDI Udev rule for the USB port that they're plugged into not that you've got. So there are caveats but they still work way better. One thing which in our lab is causing a lot of difficulty is that it seems it's not possible to buy USB hubs which have individual power ports switching from a reliable manufacturer. So we've been thinking of building all the ones but we are a software company. So one thing I thought of on this that I've been thinking I have these 28 port USB hubs and I have four of them and every port has a push button on the port so I was actually thinking about just modifying one of those with yet another set of relays on top of this USB plug that could push the buttons to do that so that's a relatively cheap solution and one more pile of relays. Yeah, I have seen some USB hubs that have switchable ports but they're only like four ports and they're pretty expensive. I don't know, does anybody else have USB hubs with switchable power that work reliably? Yeah. I think that Tom Marini from Consulco mentioned one and Portland it's called Epsilon Cush. So there's one that you can control from software. Yeah, just three ports. So it's not that much but I think it's open hardware so the good thing is that you if you're an electronic engineer you can scale that up because it's open software and I think also the driver is open source. Thank you. And I think that kind of emphasises the need for good wiki for information like this just knowing where you can find this type of hardware. Page exists. Okay. Great, thanks very much. Question? Yeah, the other thing is please add your... Oh sorry. So if you go to the board farm page on the e-links would actually go to recent changes it should be at the top of the list. But if you don't mind us knowing your name and contact information I tend to anonymise my email address by replacing the at sign with the word at. Hopefully that confuses the NSA. But then if you don't mind adding your hardware that you're using and if you're using a particular set of software that would be good to know. So like if you've written your own I put down a couple like lab grid PDU client EB farm and TTC on a list there and if you have links to any of the documentation on those that would be great. So just start collecting information on these things would be great. I think we're running out of time but I just wanted to mention two projects. The Leonardo people have been building this open TAC test automation controller board. I don't know what the status of that is but basically he bought for a PCB with connectors for PCBs which you can control network, USB, serial and so on and it is basically a beagle bone shield just 19 inches wide and then there's from the chromium or as Google people there's this servo board which we basically have one board with a synchronised connector where they can just connect a chromium or as device and control everything that's maybe an interesting idea to look for. Thank you. Anyone got anything else to add before we close? No, okay. So one more plug for a talk later this week on tomorrow at 10 1050 or whatever that 1050 slot is I'll be doing a talk on what we're calling lab in a box which we've scaled down a lab basically put a PC together with a bunch of small boards inside the PC all powered off the PC power supply switched by a Bay Libre Acme Cape with the idea being to kind of demonstrate how to easily put together a small lab with all the software preconfigured and stuff with lava and working with kernel CI and all this stuff all kind of in one PC so it's not necessarily a model but it shows how to put all the software and all the pieces together for a simple six or seven boards all packaged cleanly in one PC. Just quickly. Regarding mailing list and before we start to exchange pieces of paper with emails on it so I will ask the LFIT guys to set up a mailing list, board farms justwatchlists.linuxfoundation.org Okay? Thanks very much. Okay, thank you everyone. Hope it's been useful. Thank you. Goodbye.