 Hydiwg, rydyn ni'n gweithio. Ac ydych yn ddiddordeb y cyfnod o Debyn o'r Yw'r Ywch Tŷ Prowgect, oed yn ymgylchedd ein bod arall. Beth yna'n ddweud, mae'r rhai gyfreidio'r cyfrifio? Mae'r cyfrifio sydd wedi'u gweld eu cyfrifio o'r hamlau? Mae'r cyfnod o'r ffordd o'i gweithio, oherwydd mae'n ffordd o ddebyn projekty sydd wedi'u cywethaf ymweld. Felly mae'n ddebyn cyfnod o'r ddebyn. Rydyn ni'n gweithio. Mae'n dda i'r qurtaeth. Rwy'n meddwl yma'r ysgol yng nghymru? Mae'n ddod i'r ffordd, rydyn ni'n ddod. Rydyn ni'n ddod. Rydyn ni'n ddod i'r ffordd? Rydyn ni'n ddod. Mae'r cyffnod yr ysgol yn hyn o am erinol oherwydd am y ddebyn cyffnod. Rydyn ni'n ddod i'r qurtaeth i'r ddebyn. Rydyn ni'n ddod i'r ddebyn cyffnod yma'r cyffnod ym hyn o'r llwyffy Марfodd. Rydyn ni'n ddod i'r qurtaeth i'r llwyffy Merfer. Rydyn ni'n ddod i'r cyffnod. Rydyn ni'n ddod i'r cefnod o half iddyn ni'n gweithio ddweudio. Rydyn ni'n ddod i'r gweithio. I'm setting out to build a project, what should I use? And then I want to talk a little bit about Debian and the good things about Debian. And then I'll talk about Yokto and the good things about Yokto. And then I'll try and reach some kind of conclusion here. I've got to say this is not an in-depth technical presentation. I just want to discuss the issues around these two environments. So, the dilemma. So I'm sitting down, I've got a new device to design. How are we going to build it? On the one hand, I want to have it done as quickly and efficiently as possible. But on the other hand, I want it to be robust and maintainable. So do I choose Debian or Yokto? So yeah, you can go off the peg. So essentially, if you choose a Debian-type distro, you have an off-the-peg solution. You just grab the bits you want, put it together. You have a solution almost immediately. I'm using Debian here as a kind of placeholder for any other distribution of your choice. So it could be Ubuntu or Fedora or whatever else. So when I say Debian, I just mean a generic desktop distro. The other real option then is I can go bespoke. I can go to my tailor and have my suit fitted exactly to my wonderful form. So with Yokto project or build route or a similar build tool, you have a specification of exactly what you want to go into your target system and that's what the system builds. So off the peg or bespoke. Those are your options. So looking at Debian first then. So I think we all know what Debian is. It's a fully featured desktop and server distro. Tens of thousands of packages, very stable, long-term support. All that wonderful stuff. And yeah, the package feeds we get from Debian are binaries. So we can just install a binary package onto our device. No messing around with funny cross compilers and things. So it's a nice, easy environment to work with. I'm looking at Debian from an embedded point of view. So that means I'm typically not using PC hardware. So the architectures that Debian supports out of the box that are most relevant to us are really the ARM architectures. And it turns out that Debian supports three different flavours of ARM. 64-bit, ARM64, ARM-HF for the Cortex-A, 32-bit processors and ARM-EL for the ancient and probably not used anymore ARM 9.20 type processors. And I guess that the Raspberry Pi is one of the key reasons why I'm talking about this. Everybody has a Raspberry Pi. Anybody here not have a Raspberry Pi? Oh, three people. Well, you should get one. So we all know Raspberry Pi. You can grab this thing and everybody knows how to do this thing. Raspberry Pi actually doesn't run Debian per se. It runs Raspbarian. But Raspbarian is just a reimagining of Debian essentially because of the peculiar architecture that Raspberry Pi has. Things are compiled specifically for the Broadcom chip set. But other than that, Raspbarian is basically Debian. If you're familiar with the Beagleboard boards, so that's the Beagleboard, the Beaglebone and all the various other Beagles. Yeah, they will come with a Debian distraught of the box. And probably most of the shelf, single board computers and modules will come with a Debian type distraught. So the basic paradigm for developing with Debian and building up your product set on Debian is you have a root file system, which you've got from somewhere or other. Somewhere within that there will be a sources.list file. And that will point to the repositories where you can go and install additional packages. So you do something like apt install, name a package. It goes to the Debian servers. It will find the best match for your architecture and the versions of Debian you're using. And you download this deb file, DEB. And the deb file is the binder which contains all the stuff. You install it. Now you have the xyz command available. So this is typically what happens then. So you sit down with your project specification. You want to control some lights in a room or you want to control some machinery even. Raspberry Pi, let's start typing. So typically you start off with an off-the-shelf, an off-the-peg image from Raspberry and now from Beagleboards.org or wherever. Then you strip out the stuff you don't want. So you can remove some packages because there are some services you don't need and some other stuff. We kind of slim it down to just the stuff we really need. Then you add in the stuff you do want apt install. Anything that you've written using compile code like C or C++, you'll need to compile for the target board. And typically you do a native compile on the target board. So you can just run your GCC. You'll pick up all the standard libraries. Again, that's nice and easy if a little bit slow. Plus any other tweaks, configuration parameters, other bits and pieces you need to change. And at the end of this, you have a thing called a golden master. So this is now your system set up with all the applications installed or the packages installed. Everything's tested and working. So once you have your golden master, you then take a copy of it using DD, for example. And then you just clone that on to every unit you ship and you have a product. Okay, so that is the naive view of using a Debian-based distro to create a product. And I've seen people do this many, many times over on many different products. So who can see something that can go wrong here? Good, me too. So what can go wrong? The problem is this golden master thing. Typically it's not well documented how the golden master was created. It's kind of done on the fly with some command line stuff. Maybe you script it, but even so there's some steps that won't be documented. So it becomes kind of difficult to update this thing. You can do incremental updates easy enough exactly to do on your laptop. You can do apt upgrade. You can do apt install. You can do tweaks like that. But if you want to do something really major, for example, if you want to switch to a different, a later version of the operating system, you have to start over from scratch and reinstall everything from scratch and you can't exactly remember what you installed in which order. So doing major changes becomes either hard or just very, very hard. The most subtle problem is that probably the golden master was created by an individual or a group of individuals. They will have left some kind of footprint or fingerprint on the system. So maybe there are some user accounts and passwords which they haven't cleaned up. Maybe there is the bash history file which has the record of all the commands they ever entered which could be useful when you're regenerating this, by the way. Maybe there will be some old system log files lying around. It's difficult to clean up everything when you've done this kind of way. So you end up with something which is kind of not entirely clean. So if you really are serious about using distro like Debian for real products, you kind of need to go back and do it properly. So that means that you need to write robust scripts that are going to create the image from scratch using Rootstock or Debootstrap or Linaro image builder or whatever and create the root file system image from scratch. So now you're going kind of semi bespoke. So I'm going to build my own root file system image from a bunch of packages, just the ones that I want. Then I import my own software configuration. So I cross compile my own applications, install them, install the configuration stuff. Ideally, if I want to make this really reproducible, it would be really good to take my application and build it as a dev file, a DB file, then I can distribute my application along with all the other DB files. So that's kind of the better way of doing it. Now you have a reproducible thing. You should be able to, at the touch of a script, be able to reproduce exactly that same image. So now doing updates is easier. If I want to switch to a later version of Debian, I can just make a few tweaks and I should be able to rebuild my system without too much effort. So now I have a clean copy of Debian, not one that somebody has fiddled around with for several months. So there are good examples of this. If you look at the way that the Beagleboard people do this, there's the reference there to the Beagleboard image builder. So that's how the Debian image for Beaglebones is generated and other Beagles. And likewise for Raspberry Pi, you can go and have a look, for example, at PiGen, and that also is a nice little build tool that would generate a clean Debian image from scratch. I just want to say a little bit about software update, by the way, because this is a beat I've had in my bonnet for a good many years, one of the apparent attractions of Debian is that you have out of the box software update. So you can just do apt update and it will update everything. I just want to say that this is not always the solution you think it might be. The major problem is that the apt command is not atomic. In other words, if your system loses power during an apt update, there's a good chance that you'll end up with a corrupted system. So really if you are doing system updates on Debian systems, particularly out in the field in remote locations, I just want to say apt update is not really the answer. You need something a bit more sophisticated. So Debian is now, or the off the peg approach now is looking a little bit less attractive. It's a little bit less off the peg because you're having to create some scripts to generate the thing from scratch. And even so, you still end up with something which isn't entirely embedded friendly. So it's going to be large compared to what you can do with Yocto. And the main disadvantage of large, apart from the fact that it's going to take up more storage space, is that you're running more software which means more attack vectors and therefore a less secure system. The fact that Debian only has a limited number of architectures can be a problem, especially when it comes to ARM processors. There are a vast range of different combinations of features on the various ARM processors that you can get. When you compile Debian, then you're compiling really for a lowest common denominator. So that means that the binary, the dead packages you're installing are not optimally compiled for whatever architecture you have. And that can have some impacts, particularly if you have co-processors and such like, which are not being used by the main Debian archives. Debian doesn't really know that much about flash memory. It's not optimized to reduce flash memory rights. And as you may know, one of the problems with flash memory is every time you write to it, you damage it a little bit. And typically you can write to each flash sector about 3,000 times before it comes unusable. So you want to try and reduce the wear on your flash, typically. Debian doesn't really have any options to do that. And coming back to this native compiling thing, one of the big downsides is if you've got to compile everything on the target board, these target boards generally aren't particularly powerful. They don't have that much memory or storage. And so compiling stuff natively can become a pain. It's slow and, yeah, is a pain. And even when you've done all that, I haven't really spoken about the kernel and the bootloader. Debian, of course, isn't going to supply kernel for your particular board because that's going to be specific to your board. Likewise, the bootloader is something very board specific. So I'm just talking, when I talk about Debian, I'm just talking about the root file system. You still have to provide your bootloader and your kernel outside of this somehow. So that's the Debian side of things. Let's have a look at the other thing. Let's have a look at Yocto project. Stroke open embedded. So I apologize to the open embedded people in the audience that I'm emphasising Yocto project and not open embedded here. Let me say again that, as with Debian, which is a placeholder for any distro, when I talk about Yocto or Yocto project, I'm also implicitly implying, implicitly including the open embedded stuff because that's what it is. So the thing about open embedded stroke Yocto project is that you basically give it a list of instructions and it generates the distro for you. So you can customise the distro to have exactly the bits you want, compile the way you want, et cetera, et cetera. So the key difference here then is instead of taking binary packages and installing them, we are taking source code and compiling that source code to get the package for our target system. So Yocto project has very good support, pretty much industry-wide support in the embedded industry. So ARM, MIPS, PowerPC, X86 all have their own Yocto support layers. Likewise, anybody making single board computers or system on module devices will almost certainly have Yocto support. And there is also, if you want to outsource stuff or get commercial support for various things, there's a whole bunch of companies who specialise in this area. OK, so looking at the Yocto way of doing things, this slide here shows that we have this stuff called Metadata, which I'll come to in a moment. And you have this command called BitBake. So BitBake reads a thing called a recipe from the metadata. You give it the recipe name. It will then go to the upstream source code, download the source code, for example, a tar.gz file, compile it, using a cross compiler, generate an RPM file. So now we have xyzrpm. And then as a final stage, there's a thing called dorootfs, which will take all the RPMs you want and generate the root file system for you. So that's the steps are longer, more complicated. But the end result is we take source code and we generate a root file system. So the power of Yocto project is, yep, it's all in the metadata guys. And in this slide I just want to illustrate three particular types of the metadata just to give you an idea of the flexibility that Yocto gives you. So there are three key parameters when you do a Yocto build. Distro, machine, image. So the distro is a bunch of configuration metadata which says what distribution do I want to create. In other words, how do I want to put my system together? What initDemon do I want to use? What SSHDemon do I want to use? And various other policy decisions. Then there is the machine. So the machine is what target hardware am I building for? And I should be able to change that for a given distro. I should just be able to switch machines from an ARM-based machine to an x86-based machine to a MIPS-based machine. I should still get in each case a working system that looks exactly the same but just runs on a different architecture. And then the third leg of the stool is the image. So when I bit bake something I give an image that I want to build and it will then build the... So the image is just basically a list of packages, ultimately a list of the RPMs. So it should be able to cross compile all those RPMs, assemble them into a root file system for the machine using the distro configuration that I have. So that's the good stuff, obviously. So now let's admit that Yocto has some downsides. Probably the one that most people will initially come to is the learning curve. So whereas with Debian I can just do apt install and everything works apparently. With the Yocto project you have to understand how this metadata works and it's quite complicated and you have to invest some time and effort in getting this thing set up. So seat learning curve is probably the biggest reason I hear people say that Yocto project is a pain. Support is also an issue. So whereas with a Debian-type distro you're going to get long-term support for maybe five years or something. Each Yocto release is only supported for basically two release cycles. There's one every six months, so that's roughly 12 months. Once you get to the end of that 12 month community support you either go it alone or maybe you outsource to one of the companies I mentioned a few slides back who can do kind of paid for long-term support. And since you're going to be building loads and loads of stuff you need quite powerful hardware to build a Yocto system. And if you're doing this on a nightly build then you need something that will complete the build within eight hours or something or six hours. So you're going to have to invest in some hardware to do this kind of stuff. Okay, so that's the overview of Debian on the one side, Yocto on the other side. I'm at the point of coming to some kind of conclusions then which do I use in what situation. So neither is best in all cases. So Debian. Debian is really really good for proof of concepts and prototypes because I can put something together really quickly. I can have something up running within a couple of days typically. It's great for kind of one-off projects where you have just a small number of units in use. So I know people who for example use Raspberry Pi on production lines and they use the Raspberry Pi to run certain tests to check the quality of the whatever. So yeah, that's nice and easy to do. You don't need any particularly robustness there and you're only going to deploy half a dozen units so you can go around and update them yourself. That's easy enough. And yeah, it works best obviously if you have commodity hardware of some kind that supports Debian type distros. Conversely then, Yocto project is great if you have custom hardware. So there is no Debian port already for this hardware and you could make your own Debian port of course but that's almost as much trouble as making Yocto work in the first place. So custom hardware. Because you're generating exactly the distro specification that you want, you can leave out the staff so your attack surface is reduced and also your storage requirements are reduced and also in the same vein the amount of memory, the amount of RAM you're going to need is probably reduced. Okay, so we can make smaller systems more compact systems using Yocto than we can using off the peg Debian. And one final feature that Yocto has and also build route, I haven't mentioned that but that works as well is that if you want to do license compliance these open source build tools generally have a feature which allows you to generate a license manifest so you can see for each package exactly what license it's using. Getting that same information for a desktop distro like Debian is not so easy. So that's it. That is totally it. So let me open the floor to any questions you may have otherwise we can go and have a coffee. So questions. Starting over there. I have a microphone. So if you actually look at doing your own Debian packaging are you already alluded to if you're building Debian from scratch it's almost the same complexity as Yocto but just to do your own Debian packaging how complex would you see that as compared to writing recipes for Yocto? So in principle creating a .deb file a Debian package it's almost exactly the same steps as creating a Yocto project recipe. The syntax and everything is different obviously but the concept is the same. So it's pretty much the same level of complexity. Anyone else in the middle there? I'm sorry I'm going to have to ask you to pass the microphone round. But I would encourage you to use the microphone because if you just yell out then this microphone here isn't going to pick that up and it's not going to be on the recording. Hi. It's more a comment, not a question. I'm a Yocto fan so I'm totally on your side but to be fair I'd like to mention here the project Albe. Don't know if you know it. It's from the company Pangotronics and they are promoting to use Debian for embedded systems because their argument is for Debian there are thousands of developers they get paid from Debian project to work on security issues and this is the Yocto project clearly it has a community and it's part of the Linux foundation but there are not so much engineers working on exactly that topic, the security. I totally agree with you. So I think maybe I didn't emphasise it enough one of the key benefits of Debian is you do have that community support and you do have a bunch of people doing security updates for all the major components which you kind of get for free. That doesn't happen with Yocto quite such. Yes and just a laughing to that the Albu project supports also a meta Albu layer that can be used within Yocto to build up a root file system with Debian packages. So they access on the Debian database with the RPMs. The name of that project was? L-B-E-L-B-E I can give you a link later on. Okay cool thank you. Thanks. Okay anyone else over there? Thanks for doing the running Tim. I just wanted to add that there's a project by Calabora called Debos which is you describe the Debian root file system you'd like in YAML add the packages you want, custom scripts you want to do and it will spit out any Debian supported architecture root file systems or Tartballs. It's just a really cool project. KernelCI is using it to create the root file systems they do their boot testing with. Okay that's cool. There are many more ways than I realize for creating Debian root file systems. That's great. Just a comment on this L-B and Debos and I think that's one of the when you try to look at building a Debian embedded system there are too many ways of doing it so you have these Debos L-B there's 10 other tools that you can choose from so it's kind of hard if you're thinking long term or multiple hardware and multiple architectures that's one of the biggest big benefits of the Yocto project as I see it that you can kind of think long term in terms of hardware support and then these tools it's very fragmented. Yeah and that did actually occur to me when I was putting these slides together that when I do a quick Google for tools for creating Debian images a bunch do turn up but there isn't one that's kind of the way of doing it so you're right so there are lots of different smallest projects doing this kind of thing there isn't something with a push that say Yocto has but you know your mileage may vary it's still a valid way of doing things. Okay any more? Okay well with that then thank you all very much for being here and enjoy the rest of the day.