 Today I'm speaking for myself in sort of personal capacity as a Linux maintainer, rather than anything to do with arm. So, yeah, today I'm here to give you a whistle stop tour of embedded Linux. So, so the goal here is to give you an overview of the embedded Linux landscape and point you in the direction of some interesting projects are out there and give you some ideas for the sort of things that people do with Linux in the embedded space and how people work with Linux in the embedded space. Because it's covering a huge area, it's going to be a fairly or a very high level talk, and it's going to cover each individual area that I cover in quite a superficial level. If you have questions as kind of says please don't hesitate to ask, we can go into things a bit more deeply. The idea with the presentation is to start discussion and give you ideas. So, by all means, you know jump in whenever throughout the presentation. If you, if you have questions or comments or anything like that. So just to give sort of a brief orientation. I'm going to start off by just firstly defining the ground explaining what I mean when I'm talking about embedded systems. And then I'll move on to give a high levels tour of embedded related projects in the next space. So, I'll go to that in roughly three levels I'll start off talking about embedded projects or tool systems, the, you know, a complete thing. And then I'll drill down a bit to talk about the operating systems people use in an embedded context on Linux, or using Linux rather. I will go go down another level to talk about some of the individual programs that those operating systems use. So the Linux kernel is one example of those but there are, there are a few others. So, yeah, like I say very high level talk, mainly pointers to useful information projects and the focus here is on free software so I've skipped over a few things that are out there and use Linux but are mainly proprietary. So start off with what we mean when we talk about embedded system. Frankly, it's kind of a fuzzy concept. You can write formal definitions. But any formal definition that you write there's going to be places at the edges where, where the definition doesn't apply either something sort of fits in the definition but nobody looking at it would say it was an embedded system. There are things clearly to everybody concerned to look like embedded systems, but they don't fit the definition for whatever reason. So, what I'll say is that for the purposes of this talk is, is kind of fuzzy, but it's about how developers and users interact with the system. From a development point of view with an embedded system, you're thinking about hardware, you're thinking about very specific hardware, a particular, a particular machine, not not just a device within a machine, but a system as a whole. And you're thinking about that at a very low level interacting with the hardware, what very specifically will that hardware do. And from a user point of view, you're thinking about something which you probably know that there's a computer there, but you don't really interact with it as a general purpose computer. You interact with it as some level of fixed function device, which does a thing. And some of how it does that thing is, is based on software. But it's not really important to how you deal with the thing that that's the case. You're dealing with the system as a whole. And the development side of that gets influenced by, or the development side of thinking about things gets influenced by meeting the user that user need. And the user side of that is partly a function or. And the user side of that is partly a function of what the developers are doing. So, hopefully that makes some sense. I don't know. Please. So, when people when you say embedded system. The first thing people often think of is the small single board computers that are commonly are quite widely used. So here's a couple of examples I've got one here physically as well to give you an idea of how big these things typically are that can be much bigger. So these are small Linux, these are small systems which can run Linux, and they're complete computer on a single small board. So they've got USB, so you can plug a keyboard on a mouse in they've got ethernet and Wi-Fi so you can connect them to a network. And they've got all it's not so visible in these pictures here HDMI so you can connect up a monitor so you can deal with them just as a computer. But what's really interesting about these for the point of view of them being embedded is is the connectors they've got that allow you to plug extra hardware onto them. So if you look at the back, I can't point to this. Can anybody see my my pointer on the screen. Yes, we can see that. Oh, you can see my point. Oh great. That's very helpful. So yes, if you if you look at the this connect this here, you can see that there's a bunch of pins that you can plug something into. And there's the equivalent thing on this board here. Down there. I'm just pointing out just now. So these bring out a bunch of electrical interfaces that the the computer has. And what people do what people can do with these that really make the difference with these things being embedded systems is you can build extra hardware that plugs onto onto these systems. So for example, onto onto these connectors. So for example, this board I just waved into the camera earlier. This board actually has a small module plugged into plugging into the the equivalent pins on it on it. This is a Raspberry Pi by the way, original Raspberry Pi. And so that this small module extends this Raspberry Pi which doesn't have any audio functionality normally to have what is according to the manufacturer is a very high quality audio thing. And this is a common way of putting together a computer for some sort of fixed function fixed function purpose. It gets used a lot by people working themselves just as hobbyists, but also is quite commonly used in industrial applications. So it's a lot easier to usually to design the custom hardware you need for your application. Then it is to design a complete system. So what a lot of people who need need to want to make hardware do is they take one of these systems they design just the hardware they need, and then they use the off the shelf computer that's available on the market. And sometimes that's not what you need you need something you need the entire device to be built to your specifications. So people also build complete Linux systems from the ground up. Everything from the silicon upwards. So here's a couple of examples over on the left here we've got an Android smartphone. This is quite possibly the most common embedded Linux system that's some people have dealt with I imagine a lot of you have one of these in your hands already. And over on the right here we've got the Steam Deck. This is a gaming console that runs Linux, but the users of these systems never really deal with them as Linux systems on Android. You're dealing with the Android operating system you're installing apps on it but you don't really have general control over the system. And similarly on the Steam Deck you install your games on it, but you do in normal use you never encounter anything to do with the underlying operating system. It's all just a games and a system for loading games. However, you can go to the other extreme, you can have something that people deal with as a system, but there's basically nothing custom about it. So this is an example of an embedded product I worked on many, many years ago. This is, well, its function is to be a phone exchange for an office. The users of this system, they'd buy this box, they'd install it, and they would interact with it via a web application that the thing ran. But if you open up this system, what you'll see inside is that it's actually an off the shelf PC. The only custom, or the only interesting bits of this PC are that there's a PCI card sitting here. And this panel, the labelling is custom. But otherwise it's just a PC, the same sort of PC you would get in the store more or less just in a different case. But because of the software running on it, it's turned into an embedded system from the perspective, at least from the perspective of the users and also frankly from the perspective of the developers. And even more to the edges of embedded systems, you've got modern laptops. So a modern laptop is a general purpose computer from the point of view of the user. However, it's also a very specific piece of hardware that needs to work in enabling that hardware requires the developers to supporting it to understand the specific hardware they're dealing with. So they need to know, for example, how many speakers there are, where the speakers are, they need to know not to make the speakers so loud that the case starts to rattle or anything like that. And for some of the more advanced features of the displays and of the audio, you really care about the physical system itself. So from a development point of view, while users are dealing with this as a general purpose computer from a development point of view, the people working to support these systems are basically doing embedded development, you know, in terms of how they think about things and how they deal with things. So does this make sense to people? Are there any questions so far? No questions at this time, Mark, in the chat or Okay, feel free to ask questions. Everybody. Right, so I saw some comments going fast. They're a bit fast for me to keep up with So Sorry, hands are going up. If you want to take one or two questions, let me see who has their hand up. Alexander, if you have a question, just unmute yourself and ask. Oh, okay. I guess. Yeah, if you have a question, just ask. Or type it in the chat or Q&A. There is a hand up. Oh, sorry. I think I clicked back. Sorry. Okay, nobody. Okay, so there is. Okay, there is one question, Mark. It's a longer question. I can try to Read it out to you or it's in the chat. Would you like me to read that out to you? Yeah, that's fine. I can somewhere. So this is from Majid Who's Who's saying that there is a lot of challenges with Assemble because there's a lot of software on a typical Linux system needs to be assembled and Developers need to understand what they're doing, select the correct version of the Colonel and Develop support for their the features that are unique to their system. So he's asking if the problem lies in the choice of system. So yes. So yeah, that is certainly a big A big part of the chat. I wouldn't say it's the only challenge by a long stretch, but yeah, yeah, working out Where to start from is definitely a big challenge that faces people Working in with embedded systems that you Often what people do is they look at What other people are using for similar products and systems and they do something similar. Often there's what are called reference designs. So these are No, I don't have one to wave the camera right now. So these are systems that chip manufacturers or other people put together They're intended to provide an example of sort of how you could make a system. They're not they're not properly finished. They've usually got a lot of rough edges. But they're intended to help people with the problem you raise of figuring out where to start from. Yep. So you're saying in summary, your question is how to choose the right system to start from yes and this this concept of the reference system is Looking at what other people are doing in the same space is often a good place to go with them. You take something that somebody sort of integrated maybe 90%. And then you customize both the hardware and the software to make whatever is you want to do. And if you're doing something truly new, then the Completely different to everybody else, then the problem gets a lot harder. But usually you're not dealing completely with The green field usually you your system is based on something Either in hardware or software terms. And so you can then take take a look at how whatever it was your system is built on was It was put together and you can use that as a starting point and customize it. And yes, as an angel says in the chat, don't reinvent the wheel. So I see there's also a question from Salah. Hopefully I've not mispronounced your name too badly. Asking if users can take a backup of Files and embedded system to somewhere else. And that's something that's really depends on how the The person or the people making the embedded system have built the system. Some systems Are built in a way that would allow that some systems aren't it's really a feature of the the system. And some systems don't even have any enough state on them to have files you might want to back up. So Yeah, it's That's So the question just vanished from me. So yeah, so that that that's A thing that could be possible in some systems but isn't necessarily guaranteed to be available. It's some it's a feature that the system would have to have that the The developers would have to implement. Okay, it looks like we have another question in the chat from Fabio. Do you have a scorecard for the most popular and or promising platform. So Not Not really so I mean I will like a lot of what the rest of the presentation is frankly is going through and walking around a bunch of different Different platforms. I think that's That's not really a question that it's It's possible to Answer you answer because different platforms are designed with very different purposes in mind. So for example, a A system that's built to be a mobile phone will be very focused on things like power consumption and for modern phones where people expect fancy graphics and High-speed networking, you know that those will be, you know, key features that people care about enabling. But on the other hand a system that's designed to run, say an industrial process or control the elevators in a building or something Will maybe be designed, you know, they'll probably be plugged into the mains. They don't really care about power consumption. Unlike a phone they'll be expected to last for say 10, 20 years. And so there's a whole bunch of other considerations that come there with in terms of availability of components, reliability of the hardware That are just not relevant for or interesting for mobile phones. So, yeah, I don't think you can make an you can give a specific answer to that question. I think that's something that you have to If you're thinking about doing something in the embedded space you have to look at and consider for yourself and Work out what matters to you and then select a system that Both hardware and software that will Deliver what the things that you need to make your system work well. So speaking of systems. Let's start going through. If there's no further questions, let's start going through our Tour of some of the system level software that's out there. Here we go. Yeah, so the the the first one I'll mention is Android. That's Obviously very popular in the mobile space it's That's that's what it's primarily there for but it's not some it's not actually just for phones it's very widely used in general that that comes from That comes from a couple of Places what one is that chip manufacturers see that there's a lot of phones being sold and they want to Get into that market so they They target a lot of their software support at The phone at the phone market and then people want to use those chips and other applications they pick them up and they pick up the software that came with the chips Which happens to be based on Android. And the other thing is that while Android is intended for phones it's also just a general user interface for Custom hardware that can be repurposed with a bit of work for other things so people Already have skills or know people with skills in In Android and they Until they build their system on Android so they can reuse all their user interface Design and implementation knowledge With a familiar system and also get third party apps maybe depending on what the application is So In terms of developing Android itself rather than Developing with Android the two big areas that people work on are Device support so enabling systems to run run Android Whether that's new hardware or maybe sometimes keeping older hardware running newer versions of Android And also working on adding us features that Are needed for whatever application they have So maybe feature controlling real time maybe some fancy multimedia things Who knows The sky's limit really when it comes to think of applications But unfortunately For people not working for big companies Android itself is mostly developed While it is open source it's mostly developed within Google with their Partners for For application, specific applications So a lot of the community effort around Android is centered on Lineage OS Which is a project that takes the Open source releases that Google do from the Android or the Android project within Google does And then they build on that so they do a bunch of work in both supporting hardware So like I mentioned keeping old phones running is a big part of that And also enabling OS features that people want So one reason you might run Lineage OS is because there's something you really want Lineage OS happens to have it So Sandeep in the chat Mentioned that he's seen Android used for car dashboards, exercise machines and point of sale machines Yep all good examples Yeah I actually have an exercise machine sitting in the other room in my flat here Which runs Android And Majeed is asking for Recommendations for securing embedded systems So that's a very open-ended question When you ask about securing things So one The important thing with security is to think of what the attacks You're worried about are And working out how to both try and stop them happening obviously And how to mitigate problems if they do happen So there's big things you can do Like not connecting your system to the internet If your system has no network connection Then there's a lot less of your system that somebody can talk to And make it do untoward things If you must connect your system to the internet Make sure you can ship software updates for it And provide those updates promptly In terms of non-network things Yeah Tim Bird has linked in the chat to some security recommendations Thanks Tim So yeah in terms of avoiding the system doing untoward things Based on somebody who can physically access the system It's a lot tougher To make sure that whatever ways people can physically interact with the system Only do the things that they're supposed to do Which in turn depends on what those things are And this comes back to the backup question that was asked earlier If you're supporting somebody plugging in a USB drive And copying files onto it Then you need to start worrying about What will happen if somebody plugs in a malicious USB drive That's maybe trying to trigger some software bug on your system So that's kind of a rambling answer But hopefully that's helpful pointers at least to things And like Tim's link there Which is to the embedded Linux wiki Should hopefully have a bunch of stuff that's a bit more practical and useful And less hand-wavy And Eric has asked in the Q&As about secure distribution upgrades I don't have a link off the top of my head like Eric asked for That I could point to But I would suggest checking out the embedded Linux wiki And in particular there's a list of presentations there From the embedded Linux conference over the years And I know that there have been some talks done at ELC in the past About over-the-ear updates Which hopefully would point you in roughly the right direction But yeah I'm afraid I can't I don't have anything immediately to hand off the top of my head And there's another question from Sandeep about Driver and development and debugging So I was going to mention the kernel later on towards the end of the presentation So if you could remind me to talk about that topic Later on when I get to the kernel bit that would be great, thanks I'll do that Thanks Right, so moving on, yep So if you don't like Android for whatever reason There are a few other phone and tablet-focused OSes that you can look at One of the most interesting ones is Post Market OS Which is a from scratch implementation of something that's essentially an Android competitor It's an interesting project to check out And there's also Ubuntu Touch Which was originally developed by Canonical Although they now don't really work on it so much And it's now entirely done by the community Which is as the name suggests Basically a port of Ubuntu to work on phones Phone type hardware like tablets And Plasma Mobile is a similar idea for KDE The desktop environment It's the KDE developers taking KDE and adapting it to mobile So, yeah, those are interesting projects to check out If you want to work on a phone type hardware But you're not so interested in Android for whatever reason Another big area and probably one of the earliest areas for embedded Linux is networking So this comes at all scales The interesting thing here is that Linux has a very powerful and flexible network stack It's probably the most powerful and flexible network stack out there Certainly the most powerful and flexible one that you can readily get access to But consumer products based, and even frankly a lot of enterprise products based on Linux don't really let users get access to the Anything like the potential of the stack You can't do very much with them at all So one of the OpenDWRT and OpenWRT are both projects which take Mostly consumer grade routers, just things you can buy in the store And then they replace the firmware on those systems with custom firmware Which aims to let people do whatever they want and can With the Linux networking stack, so that could be setting up VPNs, that could be advanced firewalling That could be trying to prioritise traffic Could be separating out different devices on your internal network from each other Really it's a sky's the limit thing in terms of what you can imagine doing And what you can try and accomplish with the software So in technical terms there's two big bits to these things And they're both very similar projects by the way They're doing roughly the same thing in very similar ways There's some differences I don't want to get into Frankly I don't really understand, never mind I want to get into it right now But in technical terms there's two bits to what's going on here There's the kernel bit where people implement support for the various devices that are out there So that's things like getting a kernel that can work on the device And then making sure that the kernel works as well as it possibly can So optimising the performance of the networking hardware and anything else that's relevant For example enabling features that maybe can be supported in the hardware but aren't for whatever reason And then there's also the applications that run on the system These for both DDWRT and OpenWRT are primarily web applications So when an end user is dealing with one of these systems and configuring it Usually they will try and go in via the web interface And that web interface will then configure the Linux networking stack to do whatever the user needs Whether that's setting up a VPN, monitoring traffic, firewalling things, whatever So there's a bunch of web type development that you can do in an embedded context Yeah, in these projects And in fact I think both of these projects are actually used by some commercial routers There's some router vendors have decided that they want to ship a more powerful software than their competitors So they've gone and taken these distributions and used that as the firmware for their routers IP Fire is slightly different, IP Fire is more targeting an office type enterprise application But it's kind of a similar idea, but rather than running on a consumer router That's usually targeted at running on a PC And in many ways the goals are kind of similar, but the focus of the project is different And so the way it presents itself to users is different And they also have a birding penguin as a logo, so there's that In the chat I see there's some more questions I think it's more of a comment than a question, but yeah, if you want to take a look at that Mark Yes, that's more of a, yes that's more of a statement than a question Yep, my camera is doing something, there we go Okay, so if there's no questions on the networking type applications Let's move on to home automation, so this is quite an interesting area that's fairly new Or at least the wide adoption of it's fairly new So many of you will be familiar with things like Apple's home kit and Google's home So systems for controlling and monitoring the environment in your home So measuring temperatures, turning on and off switches, detecting whether people are present And maybe triggering actions either from a computer or a phone or Automatically based on some of the things like temperature monitoring, setting the controls for your thermostat So there's a couple of big projects based on Linux that work in this space named a variety of free and open versions of these systems So they're linked here, home assistant and home assistant and open hub In terms of what they do, they're very similar, the difference between the two is to my understanding And I'm not super familiar with either project frankly is more of a focus on how they present themselves to users So open hub is a bit more low level and a bit more focused on people who really want to tinker Whereas home assistant is more aiming for something that's more polished and end user targeted But there's still scope within each for the applications of the other, it's just a more question of emphasis in slightly different communities So in terms of how these systems are put together They're very similar to DDWRT, OpenWRT IP Fire, their applications with primarily a web user interface Some control the apps as well They can run on generic single board computers like the ones I showed before For both of these you typically need some sort of radio to talk to accessories, so for example temperature monitors or what have we So those could either be on your board, on your embedded system or maybe plugged in via USB to it And often because these systems are designed to automate your home and hence be running all the time, people want to select People want to select the lowest power hardware they can to keep their energy costs down So they'll typically want to use some sort of embedded board for that reason as well They're often a bit more power efficient than a desktop computer is So I see there's a question in the chat which There is a question Mark and then I try to answer it just expand on it if you, I think they're asking for it So yeah, yeah so Angel's asking if there's a reference to map, a reference to map in match embedded systems Boards to the sources in the kernel And she was saying that vendors do sometimes provide a repository, which Yeah, so I'll cover this one a bit later when I talk about device tree I think I mean the short answer is the not really Not really one central thing usually you're better off looking at the starting from the documentation for your particular board But you can sort of trace things through from the description of the hardware but I'll cover that later when I mention device tree So yeah sure, I get if you could remind me about that if I if I forget Yeah, yeah I'm on it I have a couple of questions I'm taking notes on to remind you Great thanks Great, so if there's no questions specifically on home automation, then the last area that Let's go and mention for where there's there's full system for it's that you can get is storage so it's an this is another one that's People have been working on with Linux for a long time so again as with networking Linux has a very advanced storage and very flexible storage system and there's lots of support for network protocols as well so People will Often try and build systems which which take the Linux storage stack and make it available over the the network in various forms. Three of the big ones here are three big open source ones here. So there's a rock store, which is a generic Discs shared disks on on your local network focused product or system. So that's Again, as as with so many of these things it's primarily a web application. It also runs networking file systems software as well. And it runs on a variety of hardware from off the shelf PC hardware to to embedded boards. That's, and as well as well as just sharing files. It's also grown some Some support some support for other related things. Things where you might want to where you have a lot you might want to access the bells in the store on your storage. So things like streaming media is a big part of that. And doing backups things like that. It's not just about storage. So Open media vault is kind of similar. It's very much more focused on storage on streaming media though. So things like cataloging your movies your Audio files and then providing mechanisms to get at them and Find the media you want and view it. And finally enterprise storage OS ESOS. That's Very very different thing that that's more much more targeted at I suggest enterprise it's about providing storage technologies like ice guzzy, which does have free open implementations over the network using commodity or relatively commodity hardware. So that that's less interesting to the average home user, but equally well it can be fun to play with. So I see some more questions have popped up so machine is asking if there's resources regarding Linux in biometric systems. I am sure there are. I don't have any offhand. Maybe Tim will be able to jump in with some. Andrew has jumped in and linked to a project called fprint which I know nothing about. But yeah, I don't know but yes I'm certain there are. And pop has asked What my opinion is on using rust versus CNC plus plus. I think for us looks fun. I think I think it's a good choice. It does help with some things there's a learning curve though and it's relatively new. I think thanks to has also linked to some other some other software as well. But yeah, yes, I think it's an exciting technology that's worth and definitely worth investigating but there's a lot of new ground. I don't want to cover so I wouldn't necessarily read completely reject the use of seen plus plus just yet it's a bit early days for that. And Chien chung again apologies if I'm mispronouncing names here is asking if there are projects around cars and car entertainment systems. So that's definitely a thing people do commercially. I am not aware of anything that's generally available to sort of random people not making cars to work on. There probably are especially for the entertainment systems. But I I'm not aware of them. I can't provide points. I'm afraid sorry. If anybody else in the chat has any ideas, please do link them. That'd be very helpful. So, and yeah, Alexander is come to provide a bit of explanation to what print is as well. That's that's good. Thanks, Alexander. Moving on to the software software distributions. So one thing you can do when you're putting together your embedded system is you can just use a standard desktop Linux distribution and run it on your embedded hardware. This is often a really good choice. It can it can work well. A lot of the single board computers do ship images for say Ubuntu or Debian or Fedora. The big advantage with the standard desktop distributions is for most developers is familiarity. So you can you can run your software in an environment that's very similar to what you've got on your laptop or your desktop or whatever is you use as your daily driver computer. And you also have access to all the things that have been packaged for that distribution in the same way that they're packaged when you use them on your when you when you use them on your laptop or desktop. The disadvantage the disadvantage is that you you're dealing with a distribution that was put together primarily for use on laptops, desktops or maybe depending on the distribution servers. So sometimes you find that things don't fit very well to an embedded context. So for example, you might want to start up a GUI, which maybe your system doesn't actually have a GUI. So that's that's an issue and maybe the system isn't so easy to use without the GUI. Or maybe the system is just the the the OS you're familiar with is just a bit too big and heavyweight for the small board you're trying to run on. Yeah, in the in the chat. Prasad is asking for some from guidance on getting started. And she says starting small with Raspberry Pi is a good idea. Yeah, I fully agree with sure there. Let what one of these small embedded boards. Some of them are expensive but there's an awful lot of them are quite are quite cheap and easy to get hold of relatively speaking. So yeah, they're definitely a good place to start in terms of hardware. The other thing I think really helps with this sort of thing is finding a problem. You want to solve or a goal you want to achieve and working towards that. So, as you can see from the the area I've covered already in terms of the the systems. And seeing embedded Linux is, you know, it's a very broad and poorly directed topic. See, and it can be easy to get lost going down different alleys looking at different things unless you have a particular goal in mind. So if it works better for you, it can really help to think of just to think of something you're trying to accomplish, you know, maybe sharing files from a and as would be good or maybe something like some basic home automation stuff, you know, monitoring the temperature in your house or something. So that might be, it might be an interesting project to get started on just, you know, thinking of something you want to do and trying to accomplish that goal can provide a lot of good direction in terms of, you know, when you're trying when you're trying to get going and trying to get a grasp on things. It's a very big area. So finding something, finding this small bit that you want to look at can help focus thing and help help focus your work and help focus your learning. And Sushavan again apologies if I'm mispronouncing names is asking if the embedded system support is available in Kali. I do not know off the top of my head. Hopefully somebody in the chat does. If not, I would look at their website and see if they support. You see what systems they say they support. If they say they support arms. Video is still working. How do you're working video froze for a little bit Mark. Okay, good. Yeah, I saw the video freezing. I was worried it would fall over. I was thinking that when you when the audio cut out you were saying that check called the college in excite and you started to say arm systems if they support arm systems. Yep, yep, so yeah if these. Yeah, yeah, basically check what what systems they support and if they look embedded ish. Alexander and the and can act in the chat are saying be as the smaller Intel systems might be considered embedded as well. And yeah, Kali is just you know Kali Kali is a Linux distro as can access so it's entirely possible it supports embedded systems already maybe if it doesn't live maybe that's an interesting project to work on people are looking for embedded projects to get started with. Because yeah it should be if it doesn't support medicine systems right now. I don't see a reason why it shouldn't be possible for it to work on such things. Absolutely one more thing to add is Kali Linux as I understand is a based on cell next so for people looking at security aspects of embedded systems colleague could be. If it supports embedded it might bring some of the next features. Indeed, yeah. Yep, so look at the time and better move on a bit. So just in another distribution to mention I wanted to touch on or a thing I want to touch on briefly in the distribution section is Android. So Android, like I mentioned it's widely used. In the in the meta sphere, it's it's basically a completely independent ecosystem to the rest of it Linux apart from the kernel. So, if you're working with Android you're probably not working with other Linux technologies other than the kernel. But yeah, it's not much more to say about it as a distribution just that it's yeah if you're selecting Android then that's selected your full software stack for you. So as well as the sort of generic desktop distributions there are also a couple of distributions that are specifically targeted at embedded systems. So the smallest and simplest of them is build route. This is really this is really simple, a really simple system to get going with is designed to build everything you might run on your device from scratch, giving you full control over what what goes on there. It's got a nice simple configuration. It's got a nice simple configuration scheme and yeah it's a quite often a good place to start if you want to go build everything from the ground up and really see everything that's going on going on to your system. Something that's a bit bigger and a lot more flexible is Yocto. Yocto isn't exactly a distribution. It's more a framework for building operating systems if you like. It's designed around the concept of layering. So with Yocto you combine software from a bunch of different sources, some of which will be standard off the shelves once. So for example there's basic things with the very lowest level software like the kernel. In there there's extra layers that you can get which add support for all the hardware that you might find on a given SOC and you can also add in off the shelf layers for a bunch of standard applications and then you can write your own layer for your custom stuff and then Yocto will work out how to combine all these things, work out what it needs to do and build a full operating system image for you. That makes it sound a bit complicated. It's not quite as complicated as I've made it sound. It's very easy to get going with, but there's a lot of depth and flexibility to it. So if you're running into problems with build route not being able to do things you need to do, then Yocto is a good place to move on and look around for things to work with. It's the same idea as build route. It will build everything you want to run on all the software for your system from source. You can see exactly what's going on there and you can tinker with everything on the system. So Sulla is asking about dedicated groups specifically focused on beginners. What I would say is probably a good place to start is the any of the projects, or each of these projects will have their own community of some kind, whether that's mailing lists, IRCs, channels. Some of them have Slack and Discord or things like that as well. So you're finding something you're interested in learning more about and looking at the resources for that. That project is probably the best way into that, I would say. Some of them are more beginner focused than others. Some of them have sort of introductory tutorial things, or maybe suggested bug lists that are easy to look at, hopefully. But yeah, hopefully that's useful. Any more questions on the distributions or comments? Sulla is anticipating one of my later slides there, recommended Zephyr. Okay. So in Pedro is asking if you just need to build a minimal root file system, which software I would recommend. Yeah, both Buildroot and Yocto, as you say, will allow you to build a minimal system. At the small level, the results from Buildroot and Yocto are very similar. The differences between Buildroot and Yocto mainly come into play when you want to do something bigger and more complicated. Yocto has a lot more tunability and flexibility. But the cost of that tunability and flexibility is that you can get a lot more complexity with it too. So I would recommend looking at either of those projects and seeing what makes sense to you, what works for you. They're both perfectly suitable and good projects. Personally, I tend to use Yocto. But that's mainly because I've been using it since before Buildroot was a thing, rather than because there's any specific reason to use it. And Majeed is asking if there, I'm not clear what your question is, if you could clarify maybe. Well, while you do that in the interest of time, I'm going to move on a bit and start talking about individual projects. So obviously this is a Linux foundation thing. We're here to talk about embedded Linux. A big part of embedded Linux is the Linux kernel. There's two broad areas where people work on Linux in a specifically embedded context. One of them is hardware enablement. Can act asking if an industry, Buildroot or Yocto are used. Yes, both of them. They're both used commercially. So yeah. So probably the biggest area where people work on Linux and embedded context on the kernel itself and embedded context is on hardware is with hardware enablement. So that there's several layers where that happens. So the simplest layer is just describing the hardware so that Linux knows how to work with it from. So Linux can run on it without making any changes the actual code of Linux. So that's a thing called device tree. I will cover that a bit in one of my upcoming slides. So once you've described the hardware, you need drivers that can run on the hardware. So that's the the other big area people work on. Shiny new hardware often needs shiny new code to make it do things. So people will. People will work on that either. Either starting from scratch based on documentation from the vendor, or sometimes people will provide or people will look at the software that the vendors provided either, you know, a system, some new ship to system and has made a legally course code release or perhaps the Silicon vendor ships and reference codes that you can look at. And they will work out how to make upstream Linux work on their system or, you know, their particular Linux version that they happen to have on their system work with the hardware. And sometimes it's not just a question of doing the device specifics. It's not just the case that you've got an Ethernet controller and you just write, you can just write the standard Ethernet controller integration. Sometimes the hardware has some shiny new feature that no hardware support and Linux has had before. So you have to go and implement features in the higher level of the Linux kernel to support, to support whatever is your hardware can do so limits can understand the concept of what it's trying to accomplish. So one really simple example of that. I dealt with recently as in relatively recently as an upstream maintainer. The flash ships that a lot of embedded boards used to store their operating systems. Keep on getting originally these were usually used a single signal to transmit the transmit data from the chip to the CPU. Which was slow because you needed to you need to send every but every single bit of the image one after another. So flash vendors started adding support for sending to to for or sometimes even eight bits at once. The kernel didn't really understand how to do that so we needed to add options. To the system for talking to the spy subsystem which talks these chips to allow it to tell the spy control in the CPU and the flash chip that they should communicate with a higher number of data lanes at once. So yeah, so that's hardware enablement. And the other the other thing that people often work on is the hardware all functionally works. But maybe it doesn't work fast enough or maybe it's too power inefficient or what you know there's some there's some metric. The system isn't doing very well on and you know that the hardware could do it you can see that the system is being inefficient, but you need to. You need to do some software work to make sure that the kernel actually works well. So people will go and they will try and optimize the kernel. They will try and enhance the way it deals with hardware to make the kernel run more effectively on their heart or whatever the whatever the metric they need to hit is So there have been a lot of questions just there and I Mark they're mostly there aren't too many questions they're mostly people talking to giving information to each other. One is what are the advantages of building kernel with build route instead of building it from source. That's probably one question there that could be answered. Okay, yeah, so there it's I mean build route will build the kernel from sources part of part of what it's doing the the main advantage of doing your kernel build from within build route rather than independent independently would Is usually the build route will make a whole system image that you can just write onto an SD card or flash once your device or whatever it is you put on there. So if you buy If you if you build thing if you build the whole system at once then you you it's just one step to write that out onto your your device. You can also find that there's some software which will be able to configure itself more optimally if it knows exactly which kernel it's running on so there's maybe features that depend on kernel options that are turned on or turned off, for example that can be selected. So if you're if the things building the applications and the user space knows exactly what the kernel it's work it's supposed to be working with is it can configure itself. Or it can configure the rest of the software to work optimally with the the kernel you're building. Is this a good time for you for me to remind you about the two questions that you wanted to answer during this time. Yes, I know there was one about device tree. Do you want me to just cut and paste them in the chat. Yeah, that works. Yeah. Okay, that's the first one. Right. So, yeah, techniques techniques for developing and debugging, debugging drivers. Yeah, that's, that's a tricky one. It does depend a lot on how easy the the hardware you've got is to work with. As you say, or as the person asked the question originally said the printf debugging is or printk in the case of the kernel I guess is unfortunately quite common here. The, yeah, it's thanks for the reminder can this. It's, it's a difficult question which depends very much on the specific system you have the specific problems you face. If you can get log messages out some way that's a good. You can you do that you can either trace what's happening or you can put checks in the code to make sure that things that you believe are true at a given point in time continue to be true. And then you know display if they're if it finds a problem. You can maybe put some of these store log that diagnostic information on a disk, although there are challenges with that when it comes to making sure that the things actually written out before things explode. You can often get gdb if you've got networking or some things you associate connecting with the debugger is possible. Often even even if it's not very easy. In general, though my advice is, or you know my techniques tend to to be. Yeah, it tend to be that sort of printf asserts in there makes you know verify that everything's doing what you expect and think really hard about it it's not easy if the hardware isn't very well designed for diagnostics which. A fully integrated system often isn't. I'm afraid there isn't anything really good or straightforward there the actually the other thing to check out there is the F trace subsystems so that there are facilities for putting hooks into the kernel for adding extra logging with very low overhead, which can be really really useful so you can trace huge volumes of information with very low overhead without disrupting things in the way that you can if you're printing to say a serial console that's quite slow. Yeah. So moving on to the hardware description bit and. Yep, which is just reminding me. So the way most embedded Linux systems describe their hardware to the kernel is with a special language called device tree. I don't have the time to go into in huge detail here. I will just give you a brief overview and some which you can go and check out check out the documentation for it and. If you want, if you want to learn more, but what did what the device tree does is it writes you write out a description of what the hardware in the system is and how it's connected. And then the kernel looks at that description and then it loads drivers based on what it sees. So that's. These device tree descriptions are often the best way to match up the drivers in the kernel with the. The hardware on the board. And so you'll see. Is talking through this. This example here which is for the GPU on on the Samsung phone socks it seems. CC this. It's just saying this is a GPU and then this this line compatible here is the key one for matching drivers. So if you. This this says that. This device is can be described with a string Samsung Exynos for five four three three Mali and arm Mali T seven sixty so. When the kernel runs it will look at these compatible strings and it will go through its list of drivers. For any drive, see if it's got a driver that says it supports the use of those compatible strings. And if there is one it will try and load that driver against the. It's run with this hardware block. So if you've got a device tree for a board, you can look at the compatible strings for you see listed in that device tree. And you can then go and look at the kernel and see what's supported. And then when the driver and binds to the device, it looks at the the rest of the description of the device. So here you see this line reg says there are some registers at this hardware address that are this big. There are some interrupt signals that the. The device can use to tell the host something's happening. And then we can have other. Other properties that customize the device they'll be defined in the. On a per device basis sure has linked in the chat to. To the documentation for this I don't have time to go into any detail so. Please go and check out the documentation to see exactly how that works. One thing I will mention is that. Often. Your your system will be. There'll be a lot of different blocks of hardware that. The need the need to work together so then the device will also show links between different things. Or different components in the system. So here's a brief example. We have an MMC controller storage controller. Which needs some clocks. One of the. You can see here the I've bolded. The name. There's a named clock. Or a named. Device. With an hand in front of it. And then that references somewhere else. So. In the device tree the device that provides that clock. So here we've got this Samsung Exynos seven clock top one. Provides the clock top one is you is used by the MMC controller there. Yeah. And yeah, there's a bunch of kernel subsystems that are. Mostly used and embedded rather than. On desktop systems. So GPIOs are for simple. Simple individual wires. PWMs pulse width. Modulation. Is a sort of semi analog. Output from a. I O is for. Any random sort of sensor or analog output from a device. And the input display and audio subsystems. Well, obviously they are used on regular computers as well. There. There's quite a bit of embedded work there too, because obviously you need to interact with your system somehow. Yeah. Pedra is asking if you can load devices. After the route is loaded, you absolutely can. So I'm going to. That just works the same way as it does on a desktop system. Fully works. So I've got very little time left. So I will just briefly skate over the firmware section here. The two main. Firmware projects for Linux are UBOOT and Tiana core. Frankly, in the embedded context, you're almost certainly going to be using UBOOT. Tiana core. Is mainly for desktop systems. Although that there are a few embedded systems that. That use it. So UBOOT. Is a program that deals with the. Early setup of the computer of the board. Getting it. Working in the state that Linux needs to start. And it will also. With loading the operating system from whatever storage the system has. Getting into memory ready to start. Yeah. Again, I'm. Short on time. So I'm going to skate on quickly. And mention. Zephyr, which she mentioned before. So this is. A small operating system designed for. Either. Very small systems too small to run Linux. Or also utility processors running within a Linux system. But separately. So this is useful for situations where either your system doesn't have very many resources. Or where you need to do something with very tight time constraints. So one of the reasons. Or one of the main reasons you're likely to run into a Zephyr system. On a, when you're running Linux. Is. A project called sound open firmware. Which is the DSP firmware that's used by most. Non Qualcomm. Chips these days. Including a lot of Linux laptops. Will use a sound open firmware. To deal with the very bottom of the audio stack. So, yeah, if you have a. A small low power. Or very small system or a real time need. Zephyr is a good thing to check out. And. That's the end of the project. So. And. That's the end of the presentation and we have. Maybe. Three more minutes for. For. Questions or. Candice, I don't know if you want to take over already or. You're more than welcome to take. Answering questions. It looks like we might have just got one in the Q&A. So I see, yes. So. Is asking. What are the potential use cases for EBPF and embedded Linux systems. So. They're the same use cases as. For. EBPF. Anywhere else. It can be useful for the sort of monitoring and debugging things. We're mentioning earlier, actually. It's useful for. Yeah, it's useful for it. Yeah, it's useful for instrumenting things. It's useful for. Customizing the kernel without having to. Go and actually modify the kernel code. Yeah. Yeah, so I'm not, I don't actually use EBPF very much. So I'm not super familiar with other than the sort of trace and debug. Applications. But yeah, yeah, anything you can use EBPF for in a. Desktop system or a server you can use it for the same things in embedded system. Pedro and Ravi are asking if and when the system will be available later and yeah, Candice's said it'll be available on the Linux platform on YouTube channel. And. Yazid is asking what the main benefit of Linux is. So the, the, the main thing I would say that you're getting with Linux is an awful lot of shared work. So. I was mentioning the, the networking stack. And the storage stack earlier. You know, there's a huge amount of work gone into making sure the, or into making Linux. A best in work class. Operating system for networking and make a best in class operating system for. For storage. And. You can just pick that work up and use it in your own thing for your own purposes. And suddenly there's an awful lot of work going into making sure that the various embedded SOCs out there work well in. With Linux. And by, by taking Linux or by basing your, your work, your system on Linux, you can take advantage of all that other work that people have done. And you really only have to deal with the things that are special for, for your system. It's all the other great thing about Linux being open source is that, or specifically due to use the open source is that you have full access to the whole thing. So if you run into a problem. With enough time and effort, then hopefully you can fix it yourself or find somebody you can hire who will fix it for you. So with proprietary systems, you're often stuck waiting for a vendor to fix something for you if you run into a problem in the proprietary bits. By basing your work on an open source system like Linux, you have full control or more full control in your much less at the mercy of external vendors. That can be really important, especially if you have to support your product for a very long time. If it's one of these embedded systems that's going to be deployed for 10 or 20 years, then knowing that you don't have to rely on a vendor still being around in 10 or 20 years can be a very reassuring thing. So yeah, hopefully that answers that question. And I see that we're over time. So I should probably hand over to Candice to wrap the session up. Thank you, Mark. This has been great. That's from my side. Thank you. Thank you, Mark. And thank you, Shua, for your time today and thank you everyone for joining us. As a reminder, this recording will be on the Linux Foundation's YouTube page later today. And a copy of the presentation slides will be added to the Linux Foundation website. We hope you are able to join us for future mentorship sessions. Have a wonderful day.