 All right. Hello everybody. My name is Chris. Today I'm going to be presenting to you a talk called Behind the Curtains of making real consumer devices using Debian. I'm an engineer at Collabra. I specialize in the integration of boards and integration of customer devices into sort of Debian systems, continuous integration of those systems and kind of packaging software, that kind of thing. So if you have any questions about those sorts of things, feel free to ping me using the email that you've got on the screen at the moment. So basically this presentation is really for people who've got some hardware, whether that's development board or a final board. They've got an app running and they're missing the piece in the middle, the kind of software stack that runs everything, the embedded Linux, all of that integrated together. So I'll be talking today in two parts. The first part I'll be talking about the process that I use to bring up a new board. It doesn't matter whether that's which vendor the system on shipped by or anything like that. It's kind of agnostic to all of that. And then the second part I'll be going through some specific, more specific bits that we use for some projects, really with Debian and kind of all the bits that go around that. So continuous integration of the system, how we make it secure, that kind of thing. I've also done some previous talks on DevOps, which is the tool we use to create images, specifically Debian images, but also looking at its kind of generic tools that will work for any kind of image you want to create. So we've got some patches at the moment that are being written to run for Arch Linux, so you can build Arch Linux boxes. So please do see those previous presentations for that because that is kind of a prerequisite for this. And also we've got the famous question that we're going to be answering today. We've got the hardware, we've got the app. How hard could it possibly be in between? The short answer is very hard, but hopefully the tips I've been giving you today will help with that process. So really the first thing that you want to decide on before you start hacking or getting involved with anything is you really want to sort out the requirements of what the product actually wants to do. So you want to find the SOC or the system on chip you're interested in. You want to make sure that the peripherals you care about actually work properly. They've all got the right kind of parameters involved. And also you want to be very clear from the early day what the performance requirements are of your app and the product. Because there's a lot of chips nowadays that are very slow clock speeds, so you just want to make sure that everything is going to run nicely there. I would also highly recommend getting some SOC development kits. I mean, I've been through a few dev kits, blown them up, that kind of thing in the past. So it's useful to have one and a spare if you can for each member of your team. Or maybe the spare would just be like a spare, not one spare for each. You want to get as much stuff from the vendor as possible so you want to make sure that they've got all the BSP. Usually that will be like an image that has been created by the vendor. If you're lucky, you'll also get some source code. So that is also very quite nice if you can get the source code as well as a preview image. You're also going to want to get as much reference material, so schematics, SOC data sheets as you physically can get your hands on. Also, I recommend saving all of this information in a place where everyone can access it. So we at Collaborate Use Next Cloud, we find Next Cloud is great because we can just create an account for somebody and let them log in and download the files as necessary. Also, a really important thing is to kind of get yourself a tech contact at the vendor, whether that's the SOC or SOC vendor, because they can be really useful in helping to track down any problems that you may have with any of the technical requirements or any of the issues that you may come up with. And yeah, there are normally loads of issues, so it's always good to get someone who's really hands on at the vendor. So basically the next thing we're going to talk about is what the BSP is. It is obviously BSP stands for Board Support Package, but this could be one of many different things. So, I mean, I've seen BSPs that are normally just built images. Ideally, you'd like some source codes and build scripts, Yocto layers, Android source, dump anything like that to try and get your hands on. But I do like to get a preview image as well as all of that, so that at least you can verify the preview image works. Inside all of this stuff, you'll have a bootloader, a kernel, some root FS, whether that's pre-built root FS, or maybe a script that calls something like Debootstrap to build a Debian OS, something like Yocto or Buildroot that builds everything. And then all the scripts and things that go along with that, documentation of how to get it running. Usually, with any kernel you may get or bootloader, it's usually going to be quite an old release. It's going to have some patches on it downstream that will be of questionable quality. I mean, in some cases, I've seen vendors that do really good stuff. I've also seen some vendors that don't do so good. So really, I tend to want to choose the SOC based on how good the vendor is and how good the BSP is. But in some cases, you may just be stuck with it because of time or cost concerns. Usually it's because the cheap BSP or the cheap vendor has got terrible BSP, so you're stuck with it. So, unfortunately for you. I mean, it's not as simple as that, but I think generally that is how things work. So, it's quite generic advice there really, but at least try and get as much information as possible. And like I say, the pre-built image is really what I think is the most important thing, because then you can validate that everything works without having to compile anything. What I also tend to do is document how to flash that. I mean, in most cases, you'll flash it to an SD card, but in other cases, more recently this sort of thing is happening where you flash it over a custom USB protocol. It's always good to document how to build that flash of software and how to run that flash of software to flash the board. Not only will this help later in the factory build process, but it will also help other members of the team get started quicker. So, the first thing that you're going to want to do after flashing that image is obviously try and run the board, see what happens. Hopefully you'll get some sort of shell on serial port. Maybe you'll get a display, an X11 system on the HDMI port or a touchscreen. That would be very nice if you could get that. What I then tend to do is play around with everything, verify that things all work as expected. The drivers for the peripherals that you're going to be using just kind of work okay and get things running there as well as document how to do all of that stuff. If you don't do this early, you're going to be stabbing in the dark later when you try and get things running on mainline kernels. So I highly recommend getting things running in the BSP early rather than leaving it until too late. I tend to want to spend a week or so getting the BSP running getting everything running on the BSP to try and just get things running in a dirty environment to be honest. So then that brings us on to our next step which is replacing things. So you've got this BSP image of theirs running and you kind of want to start putting your app on it and replacing the bad stuff with potentially some mainline stuff if possible or at least something that you can compile yourself. So the first thing I always do is analyze the partition layout whether that's on device or analyzing the actual image itself that comes out of the BSP or kind of just a mixture of the two really. A top tip here, you can either mount the image on your local machine using loopback devices and all that stuff which is quite pain but you could also use fake machine which is the library Deboes uses to create his virtual machine to mount the image and then you can do all sorts of scripted things on that BSP image inside fake machine. Normally what I tend to do is extract all the binary blobs from that image so the boot loader, the kernel, even the root FS I'll just extract it all somewhere and then I will stitch that back together using a Deboes recipe to actually create my own version of that image even though it's using the binary blobs from the BSP and this is quite simple really because you then have an image that you've created yourself with the grep partition layout and you can then verify that that image works just fine and the partition layout is all okay and everything before you go any further and then we tend to do a replacement approach and as a top down replacement approach so what we tend to do is replace the root FS first with something like a very basic Debian maybe Debian unstable release or something like that using Deboes we tend to use Debian because it's quite mature and the process is make sure that the system is stable and very secure the process inside of Debian so basically we use Deboes to run the bootstrap action an app to actually install Debian itself a very basic system then we'll use the apt actions to actually install packages in there that are relevant so then we'll also use the run and overlay actions in Deboes to actually run scripts inside the root FS copy files in that kind of thing just a real basic system that we can get a shell to or maybe even a X or Western system and then we'll make sure that all runs okay and everything's all happy still with the replaced root file system and then the next thing I like to do is to build the BSP kernel and other things in the BSP myself so the boot loader maybe normally this will require a holder compiler as well for some vendors so I like to get all of that done first I wouldn't do this in CI I wouldn't do this on the local desktop with excuse me with some documentation around it to ensure that anyone in the future could do the same things well if needed then if you want to get some extra credit you can replace the boot loader with the boot loader built yourself sometimes it's worth just sticking with binary blocks for the boot loader in some cases like the first and second stage boot loaders and then also you'd want to rebase the kernel patches and the boot loader patches this isn't for the faint of heart as an integration engineer myself I would really enlist the help of the kernel engineer to help with this task I would prefer to give that task to somebody else to get on with that knows a lot more about the kernel especially on this specific chip I'd also want to make sure that I chose a stable kernel release to base those patches on so really all of that is going to be quite hard work I think trudging work I call it so it's like the hard stuff that takes a lot of the effort but after you've done that you'll have a custom image so that's really good you can stop there if you really want because that's part 1 of the presentation finished but now you have that custom image the hard part is then actually making that custom image into a product so you have a basic kind of image and then a lot of people will stop at that stage and say right the product's done but actually there's a lot of things that we haven't yet thought about including continuous integration security actually packaging the application to run on the system upgrades of the system chain of trust security updates automated testing factory provisioning and lots of other things so we'll cover all of that in this next part so integration in CI I mean continuous integration is a really important thing for us at least when we're building customer projects because we have a lot of different engineers who will be working on the same product having to download all the tooling and all of the right cross compilers and things like that would be a mess so really we decide early to create continuous integration we use GitLab for it but you could use other things like GitHub or Azure DevOps Jenkins that kind of thing but we we decide to push all the scripts up early into continuous integration so you've got a really nice base to start with so yeah you can build the DevOps recipes at least in all of these things GitHub actions, GitLab CI as your DevOps Jenkins you can build it on your local machine which is quite nice because DevOps actually abstracts all of the things away so that it doesn't matter where you run it in a very similar environment each time running on a system like GitLab CI also gives you lots of other fun features so you can set up schedules to run nightly builds that are done automatically you can have merge requests so other members of the team can actually do merges and be reviewed by people so you've got like a proper process there you can add tags to releases loads of other things sort of thing you get when you have a software project in Git you can do all of the same sorts of things also the really nice thing that I like here is we've got chat integration and email integration so any time build fails the right people get notified and we can sort out the right people to get sort it out so as well an important thing to look at is security so you want to lock down the serial reports on the device this is both in the boot loader as well as the OS itself so you don't want people to be able to interrupt the boot process and write their own commands in and launch their own things you also don't want people to be able to go on the serial report and get their Linux console so it's always worth checking all the serial reports the ones that aren't actually broken out to the user so I mean most SSH now have got multiple serial reports it's worth checking all of them and making sure they're all locked down also it's worth disabling any debug services or any factory installation things SSH is a common one really common to have SSH enabled in these systems so really nice just to close all of that down make sure that no one can get in so then we want to go on to packaging your application I mean there's a big thing these days where people are packaging their apps in containers, so docker containers so they can run on sort of any machine there's also the camp who run their software still on the native side native software is much more likely to happen if you've got drivers or things inside your software that talks more closely to the hardware it's really your choice docker these days you can just install docker and everything just works really well we use OBS to build Debian packages from customer software and apps and we store the source code in GitLab so OBS is open build service from SUSE it's a very good bit of software to build things because it builds all the dependencies that the software needs in the right order and also it creates app repositories which then we use in the image building process to pull down the image so everything really is quite nice there I think so then we've got over the upgrades so obviously systems these days are normally thrown out of the door as soon as the basic kind of software features work and then we upgrade iteratively with new features over time so these days cure and validated upgrades are really important so what we don't want is we don't want someone to install a random bit of update on the system we also don't want someone to get halfway through the update and then break the device by resetting the power or something like that all of these are really things that we don't want in the past the upgrades would have happened with an apt repository or a script something like that like a cronjug with an apt but these days we can do a lot better what I tend to recommend is AB upgrades so if you don't know an AB system has got two slots in the EMMC or SD card or wherever you're storing your image each slot contains an app and the boot loader then chooses which one of these slots to boot based on the last time it started up and then once the system started your application then triggers the upgrade at the right time you know after it's connected to GSM or whatever and then triggers the reboot and the update's finished into the new slot this is very nice because you've always got one slot that works so to a user it will be as if the product will never not work as long as the boot loader handles it correctly so you can roll back to the old slot in case the upgrade fails downside to this is it requires more EMMC space but it's not really that much of a problem I don't think because storage space can be quite cheap these days you can get in the order of 16 or 32 gig systems are quite cheap now and also we can trim down root file systems to be really small in the order of around 500 meg to a gig so I don't really see that as a problem in most systems so we've been using OS tree and Raoq for upgrades I tend to prefer using Raoq because it's very extensible and it lets you do these AV upgrades or really choose how you do your upgrade which I think makes Raoq quite a nice system we can integrate both of these into devOS so the same recipe that generates the actual image that you flash to the device you can use to create an upgrade image you can sign the updates so this is very secure make sure that so long as no one can access the console they can't just install a random update from anywhere which is very nice the updates you can also encrypt the Raoq bundles as well and also optionally you can use CAsync which allows you to install chunks of an update and also only download what has changed on the system so although the image may be 500 meg you can just install the bits that have changed which may even be in the order of megabytes this is really useful when you're on GSM connections although these days those connections are really high bound with and also they're quite cheap to transfer over so it's really one of these things that you have to decide based on the application so then we get to the chain of trust so Secureboot is very useful on some x86 platforms so long as the bias vendor has actually implemented everything properly it may not be in some BSP bootloaders it might be a mess to implement so I don't really have an answer on this one really apart from going for a vendor who actually has got Secureboot with their platform all kind of sorted nicely and it's something to check in the early days really before you choose a SOC or system on chip what we like to do is we like to convert the root file system to read only this has got a lot of issues because things like system D place some files in ETC directory which needs to be right but after you've created that read only file system you can then hash it with DM Verity and the system that's being booted is the system that you think is going to be booted which is quite a nice thing to do so that no one's tampered with the operating system or the root FS you can also use overlay FS on top of these read only systems on another partition things like the configuration files so you can actually insert a configuration into ETC so that the user may need to change and if all of that stuff doesn't work so the Verity checks some or whatever doesn't actually perform at run time then you can just not load the app and then you're sure that the system has been modified then excuse me so long life time or security updates so really the main thing here is the kernel and also going back to my original point of the fact that we want to try and port your patches onto a stable kernel rather than using the BSP kernel because the mainline Linux kernel is continuously evolving the BSP kernel unfortunately never really tracks that unless you're very lucky so what we like to do is we like to port it to a long term stable kernel which is usually supported for five or more years the current long term stable is 5.10 so it's always nice to use a stable kernel because then bug fixes and security patches excuse me are flowing from the mainline kernel into that stable kernel so these stable kernels are released really often in some cases they're released weekly and so on so you could even set up an automated pipeline to pull that stable kernel and either ping your internal maintainers to actually do the upgrade or you could also do the rebase automatically as part of that that's an option so then really begun to automated testing so a collaborator we've got Lava we've got a Lava lab which contains loads of dev kits loads of custom boards and we also use that for things like kernel CI which I'm sure you've heard about which tests nightly kernels to make sure that there are no kind of regressions on real hardware which is very nice but Lava allows you to flash an image or a kernel to a board run some tests and report back whether the test's parcel failed you can set up email notifications so that if the test fails you can again send that to the right people who know what needs to be done to fix the board or fix the issue we can also submit these test jobs automatically through GitLab so when a new nightly build for instance is ran we can then run the tests automatically on those nightly builds and then providing that that works you could potentially decide to automatically push that image out to boards in the field that would need a little bit more thinking about because I think a manual test is kind of a good thing really just to make sure things are working because automated tests are only as good as the tests that you've written so when we factory provision these items we tend to set up a separate PC completely to flash images you would tend to flash the same image as you're flashing to the board and have a separate script in that image to do the flashing and then the idea is when the board turns on it would then run a custom script to do things like setting up serial numbers that kind of thing using the PC that you've got set up next to it to kind of save everything in a database so you can go as simple or as complicated as you like here yeah so really the answer to we've got the hard way we've got our app how hard because it possibly be is it's as hard as you want to make it if you do things properly in the first place the first time you do things may be quite difficult but then each product that you designed through that then gets easier and easier that was quite a high level overview of each individual pieces but we've gone for half an hour now and my time slot is only 40 minutes so if you've got any specific questions please do ask below and I'll hopefully follow up at the next conference with any specifics that may have come from this presentation or any common points if any of the above that I've spoken about sounds like something that you'd want to work on with interesting customer projects collaborate are hiring so if you follow the link in this slide you can see our job opportunities we've got all sorts up so please do take a look there and finally I'd like to say thank you and take any questions thanks