 Welcome everyone to the Embedded Linux Ecosystem BoF, Birds of a Feather session. I hope that this is an interactive session. I don't want to be saying all of my ideas. The idea is to have a discussion. There are plenty of seats here on the front row. Oh, good. There you go. It's good to be here and be in Europe, and it's good to see all your bright, shiny faces. I wish I could see your smiles, but I can't do that. I guess I'll just dive right into it. I always put the abstract at the front just because these slides go up on the E-Linux wiki, and so then people can see what the talk is supposed to be about by looking at the first slide. I'm just going to give a really brief status, what I think the status is of a couple of areas in Embedded Linux, and then discuss overview of issues, and then have an open discussion. I hope that you will feel free to complain or not complain, or praise things that you think are going well, and you'll see as we go along here. So the status. So in North America version of this conference, Embedded Linux conference in North America, I gave my traditional status of Embedded Linux talk, and I'm not going to repeat that whole thing here. It's way too long. It was an hour and a half that I squeezed into 40 minutes, but the presentation is online on the E-Linux wiki. I think there are problems with the video, so I think there might be some minutes missing at the front, but you get the general idea. But in that talk, I did some scorecards. So now, this is an exercise. You probably didn't realize you were getting kind of homework when you came to this session, but this is an exercise. I'm going to present my scorecards, and I want you to think in your mind, do I agree or disagree with the scores that Tim gave to these areas of Embedded Linux? And if you disagree, and particularly if you disagree and you think we're on the negative side, if you think I'm being generous in my assessment, then let's talk about that, right? So technology scorecard. This is, yes, I can tell from the laughter that, so system size, we're done. There's no issues on system size. Boot time, power management, real time is done. Security, still in progress. Audio videographics is in progress. Okay, so I will admit that this is based on kernel contributions in the last few years, which means, I'm going to interpret this, that based on the amount of people contributing in these areas, I think we're done, right? Because if there's no more contributions, we must be done. This is what I really think. No, system size, we're not done. Boot time, we're really not done. Power management, I don't know. Okay, Kevin's shaking his head. That's a good indication that we're not done. Real time, security is in progress. So one of the things I realized when I prepared this talk in June is that a lot of these issues are holistic, which means that in order to tackle system size, you have to attack the whole kernel simultaneously. Same thing with real time, same thing with power management. You can't just go into a driver and fix one driver, right? You have to attack the whole thing. And so they're hard problems. These are hard problems to solve. So, okay, so just keep that in mind. I'm going to keep going through some more scorecards. This is my development scorecard, okay? So build system distros. I think we're in pretty good shape. There's a lot of them. They're pretty powerful. They do a lot of stuff. Documentation, I don't know. Okay, so I'm going to, this is where I'll inject my own opinion. I think we could do a lot better on documentation. I, you know, and John Corbett is in charge of documentation in the kernel. That's the one I kind of pay attention to. It seems like it could be better. And that's not a reflection on John. It's just a reflection on, I don't know. When I first started doing kernel work, seriously, like 20 years ago, 25 years ago, I found the documentation not to be that great. And I don't think it's improved that much. So anyway, training consulting. I think there's a lot of companies doing this stuff. Toolchains, I think we're in good shape. Debugging capabilities, languages, test systems. There's a lot of work going on right now. It's kind of in progress. Hardware support will always be the bane of Linux, the fstreaming stuff. But I'm not trying to talk you out of your opinion. So keep mental notes for our discussion section. Okay, then the market scorecard. So drones, I think we're doing pretty good in drones. Robots, cars, space systems. Actually, someone disagreed with me and said, we're actually really good in space systems. So the people are shaking their heads. Routers, good. Consumer electronics, okay. Here where I will say, you know, modulo the other problems that we have in some of those other scorecard areas, we're totally dominating in consumer electronics, right? So I don't know of a television set that's being manufactured that doesn't have Linux in it. Phones were at 85% market share. And DVRs and cameras and stuff. Linux is really dominant in a lot of places. And that's just, you can go on and on and on. There's lots of different markets. Okay, so I'm just gonna rattle off some of the areas that I think that we have issues in. But first I wanna make a couple of observations. One is the paradox of embedded open source, right? So open source effects come because we can collaborate and we can share. But embedded means we're doing custom work, right? So we're customizing what we're doing. So there's a paradox there. Everyone is doing something unique and usually you're ripping stuff out, right? To hit your size goal or your performance goals. And you're customizing it because your stack is a more vertical stack, right? You're not doing a general purpose operating system. So you don't have to have every library under the sun. And so you're, and in fact, I don't know, there's a gazillion kernel config options, right? And I bet if I was to look at everyone's product in here, I'd see a different kernel configuration. Or maybe there's some common ones. But in embedded, we tend to tune things and tweak things more. I'll make that assertion. I'm sure the enterprise guys and the cloud guys would disagree with me, but that's okay. So we have a big ecosystem. I don't know if you can read this, the slides are really small. But I mean, we have tons of people who should be in our ecosystem, right? We have the Silicon vendors, the associations related to that, the design houses, I'm not sure what you call ARM. They don't fab stuff, but they're the designers of the Silicon. You have the hardware vendors. Okay, so there are people who make peripherals that go into embedded as well as the IP block vendors, right? The people who sell IP to the semiconductor vendors. There's embedded Linux vendors or embedded Linux consultancy or there's a bunch I listed on here. Oh, did you see my disclaimer? I put a disclaimer on one of the slides early saying, I really apologize if I missed your company or your market segment or your anything. So there's way too much, way too many of us to put everybody down. But then there's like board vendors, right? There's a little bunch of people that make embedded boards, maker boards and actually production ready boards. There's product manufacturers. So that's the category I'm in. I'm in at Sony, so we do consumer electronics. But then there's a whole bunch of products, right? That get manufactured, both at scale and for small units, right? So there's automotive, robotics, routers. Many, many others. There's a very long tail, tractors, rockets, underwater sensors, you name it. There's embedded Linux and everything. But then, okay, then we got industrial users. We've got makers and hobbyists. And then we have the build systems and the distributions. And then even if they're not involved in embedded, upstream developers are also kind of part of this community, right? So there are people who come to embedded Linux conference who are not embedded developers, but they come because they're part of the Linux community or part of a distribution community. And so here's my question. One of the issues that I see is, we've got 600 people here, which is really good. But there's between, estimates vary, but between 250,000 and a million embedded Linux developers in the world, okay? So where are they? Why aren't we as big as KubeCon, which gets like 5,000 people? 14, KubeCon gets 14,000, holy smokes. Okay, that's, I'm not sure I want to go to a conference with 14,000, but anyway. So here's some of the issues that I see. I see upstreaming, that's like the problem that will be with us always, but testing, some of the resources that we have, information availability, connecting with each other and ecosystem growth. So I'm gonna do this quickly, so I wanted to make sure that I have enough time for discussion. So, upstreaming, how many lines of code do you have in your product that are out of tree? And I will take a quick survey. Anybody, should we do it lines of code? I'm not sure I can count lines of code. I know how many patches that we have out of tree, that Sony has out of tree, and it's about 2,000, right? So anybody, anybody believe they have, who's in the thousands range? Thousands of patches out of tree. Okay, who's in the hundreds range? Okay, a couple of people. Who's in the tens of thousands of patches out of tree? So I used to work for Sony Mobile, and I don't wanna pick on any one company, but this was, I don't know, eight years ago or something. We had 2.5 million lines out of tree, and I won't say what the vendor was, but anyway. So, okay, so stuff out of tree is a problem, and there's a couple of reasons for that. Anyway, I'm not gonna go into all the reasons because I'll run out of time, but so all of that code that's out of tree is technical debt. As, if you're downstream from that, like if your semiconductor vendor has stuff out of tree, then that stuff that you have to maintain, you have to bug fix, and that means, likely, that you can't get on the top of tree kernel, which means that there are testing resources not available to you. There's support resource, there's the kind of community effect talking to upstream developers. It's not that they're rude, but upstream developers are more, just more focused on the current kernels, right? So if you're using an LTS kernel from 10 years ago or five years ago, then you're just not gonna get the same kind of community interactions. So, who's doing a good job, though? So I apologize. I didn't go into all the different ecosystem member categories. I picked the category of embedded Linux consultant companies or something, and these are actually pretty good numbers, I think, given the number of employees that these companies have, compared to Intel and Google and Red Hat, they do a lot of contributions, but these companies do a really good job of contributing. The question is, where is your supplier on here? These are not really suppliers. These are, usually, these are contractors, right, for some of the big companies, but they're doing good work. I bet you didn't know I measured this every kernel release, so I do take a look at this stuff, and this is just because when I'm preparing my embedded Linux status report, I like to see what's going on, so I go check what the companies have been working on. Okay, so what problems do we share that we're working on in isolation? So I know for a fact that if I was to do a comprehensive poll of the whole room, which I don't have time for now, but if I was to do a comprehensive poll, I'd find out that some of you are doing the same thing as someone in another part of the room, right, the exact same, not exact same thing, but similar work, right? You're fixing up a Wi-Fi driver, or you're wrestling with some aspect of your distribution. And so where do the problems lie? Again, I'm asking you to take mental notes because I'm gonna slow down in just a few minutes here, and we'll start having the discussion. So are there problems in the kernel? And just think about where you're spending your engineering resources. Are you working on the kernel? Are you working on the distros? Are you working on testing? Are you managing your compliance? Are you working on community infrastructure? Or do we have problems there? So where is it that we can combine and share efforts? Do we have a problem with increasing our community, or our ecosystem? Are there any issues with us bringing more people into our community? So are there obstacles? We know that lots of embedded developers exist, right? But they don't show up upstream or at our events. And what things can we do to make sure that new developers feel welcome, right? That we can increase. And are there obstacles for people to participate? So what can we do to make it so that developing, distributing, oops, and maintaining an embedded Linux product is as easy as possible? So the purpose of this BOF is really to make my life easier. But it's to make all of our lives easier, right? Through collaboration. That's the point of open source. So another question that I wanna raise is, excuse me, what resources could we utilize more effectively to build our ecosystem? And there's several different categories. Most of these are kind of communication categories. We have the Elinix Wiki, mailing list, conferences. I'm gonna go through them real quickly. Again, we will stop, I'll slow down and we'll stop and we'll go back to areas that people wanna comment on. In terms of the Elinix Wiki, who here has heard of the Elinix Wiki? Okay, good. How many use it? How many have gone to the Elinix Wiki say in the last six months to look up something? Oh, that's good. Okay, the answer is about, I don't know, 50%. This is hosted by Oregon State University, open source lab. The content, what's good on the content, it does have the repository of slides and videos from embedded Linux conference and from a few other conferences. But some areas are pretty stale. Used to be kind of more actively maintained. It would be good if it was updated. The issue, kind of an immediate issue, is that the administrator is likely to lose funding in the next year. So there actually is a paid administrator for this. I won't go into all the details, but the main job of the administrator is to do the media Wiki upgrades because it's a media Wiki based site and do spam control. And so he approves the, in order to control spam, he approves the user accounts. It's not a taxing thing, and he also rose back any spam. So you have to have a user account to contribute and so that helps with the spam a lot. So can we raise funds for an administrator? If there were ads on eLinux Wiki, would people, would that offend people? Do we have enough interest for the administration to be voluntary? Those are open questions. In terms of content, there is lots of stale content. So here's another poll. Who here has made a contribution to the eLinux Wiki in the last six months? Oh, all right. Oh, okay, you guys are awesome. I should like buy you a drink or something tonight. What? Yeah, yeah, I think if you put slides on there, we'll count it. Oh, actually, well, I could extend that if Bill Trainor put your slides on there, I'll count it too. That would be like all our speakers and stuff. So did you know that we made actually a concerted effort a couple of years ago to do a topical guide to all the slide presentations so that if you're looking like for device tree stuff or boot up time stuff, you can find all the presentations in one place. But we did this big push two years ago and then we haven't, well, COVID hit and COVID is a great excuse for a lot of lackadaisical attention to things. But so we need to refresh that. So I'll ask this at the end. I haven't made you answer except in a couple of places, but are there any volunteers to help clean up the site? That's what I thought. In terms of mailing list, okay? So did you know that there is an actual kernel.org mailing list for embedded Linux? Okay, and Linux embedded. It exists, but it's very, very seldom used. And I can think of some reasons why that's the case. It's specific to the kernel. We don't wanna turn it into a support list. That's not the purpose of it. But and for instance, this type of discussion, just the overall ecosystem discussion really wouldn't be appropriate on that list, right? Cause this is a more general type of discussion. But where could we have this discussion? The CE Linux form created something called celinux-dev that exists also, but it's very, very seldom used. And this was this intended to discuss consumer electronics Linux development. So that's an even more kind of specialized topic. So one question, should we get a more general purpose mailing list? Would a mailing list help to build our community better? Okay, so I see head shaking, no? Okay. Yeah. Okay, so the statement is what about forums? And the assertion is that's possibly a dirty word. I don't know. I don't tend to go into forums. I would rather have stuff just delivered to me. It's easier for me to like discard the stuff I'm not interested in than to actively go out to a website. I don't know how other people, if there was a forum, I could see myself getting on it. I don't know. It's hard to say though. It's like a push model versus a pull model. But would it be good to have multiple communication channels that suited different tastes? The problem is you don't wanna fragment the communication channels, right? So, okay. I'm gonna keep going. Until, oh, until 1120, a minute ago. So, conferences. And I have something very specific to ask here because we're actually making some decisions at this event concerning the future of embedded Linux conference. So we have general events like embedded Linux conference and we have specialized ones like Yaakov Dev Dave's, Automotive Linux Summit, Zephyr. And then there are regional events. I call a positive regional. I don't know if that's offensive to the positive people. Maybe they think it's a worldwide event. But we have Japan jamborees. So on ELC, very specific question. Should we split from Open Source Summit? We used to be a standalone event and then we kind of folded under the umbrella for a lot of different reasons. I don't have time to go into. But, you know, so now we're one of the conferences that's under this kind of big umbrella conference with a lot of unrelated events. And some of them like LinuxCon or emerging OS or embedded IoT, some of those are kind of related. You might be interested in, but some of them you might not be that interested in. And particularly we have an issue with Keynotes. Keynotes are, sometimes they're enlightening and they're interesting and sometimes they're just cloud people talking about stuff. And so, okay. So tomorrow you'll be happy to know there's a couple of sessions you can go to. ELC, they're running concurrent with Keynotes. But so we have an opportunity possibly to change things. So should we go, it's likely, what we're talking about is the possibility of having an umbrella event. But focus, more focused on embedded. So just be, some of those things that were on that list like Zephyr and Yachto, might be folded under an embedded event. Those seem heavily related. We might pull in other things that kind of are quasi embedded like LF Energy or there was one other one. What? Oh yeah, civil infrastructure for sure. So does that sound appealing? Okay, that's good. Yes, no? Oh, okay. Should I, it wouldn't be worthwhile sending out a survey specifically on this? Okay. Right, so okay, so I didn't get to the frequency yet, but so the frequency, right now we're doing twice per year. We're thinking about maybe going once per year. Yeah, okay. So that's a vote in favor of ELC slots parallel to the Keynotes. Okay, that's good. I've heard feedback from a lot of people that that's appreciated. So there's something for the embedded people to do well. So the other thing though, so the observation from Behan was that if we went to once a year, that doesn't mean there's only one event per year that has embedded content because there's plumbers and we might, we will likely position ourselves anti-potal, I don't know if I used that right, to plumbers. So plumbers is always in the fall. We would probably shoot for something in the spring. And so that if you wanna go to two events a year or we're also talking about alternating, so we're opposite hemisphere from them. So it would be if plumbers is in Europe, we'll be in North America. If we're in Europe, they'll be in North America. So something should be rotating through your neighborhood continentally every year. And then content or format. And this is right now. So plumbers, I talked to Steve Roestat, who's I think he's the chair this year of the plumbers. And he said plumbers is really for stuff that's in progress that has not been decided. It's for discussions about what to do next. And that is a little bit different than what we have here. We have a little bit of that here, but a lot of our content is what's happened, especially what's happened recently, so we can get presentations on that on the Wiki so that people can see. But also for new people, there's lots of content about existing subsystems or existing systems and things that people have tried so that you can see what other embedded developers are using, what's working for them, best practices type thing. So do we want it more like plumbers, less like plumbers, a little mixture of both? Maybe I'll put this in my survey. I'm way over my time, so there are other resources that maybe we could use, think about that. Now discussion time, only six minutes late. Okay, so, and then I wanna set the stage a little bit. So I have a really old mantra that I use, which is if you are spending resources on something and you know that someone else is spending resources on something, you really ought to get together with them, right? And it's like, if you could collaborate, you might not be able to, right? Because again, it's embedded, everybody's kind of doing their own thing, but you have a candidate for collaboration. So whatever it is, think about what your company is doing, where they're spending the resources, where are you personally spending your time in Linux development? And then, but also go a little bit beyond that. What do you wish was happening in the community? And maybe, you have to be careful, this is one of those boomerang things, right? Where it's a boomerang idea where you say, oh gosh, I wish documentation was better. It's like, well, it's right there, you can work on it. But maybe if as a community, we can do, instead of death by 1,000 cuts, we can do life by 1,000 cuts. We do a bunch of small projects that together don't cost us much individually, but can have a big impact for our future, right? So what are you not working on that you wish was better? Okay, and with that, I think, oh, that's the end. So I'm gonna open it up. Is there any particular topic that people wanna talk about of kind of the major issues? I talked about resources, yes, Wolfram. Okay, so let me stop, okay, yes, good. Let me stop, I'm supposed to repeat stuff and I'll never remember all of it. So just a sec, let me read. For the microphone and for people virtual, there's an issue, particularly in the Linux kernel, with maintainer gap, right? There's not enough maintainers and as we see more contributions from embedded people, my impression, and I don't wanna put words in your mouth, is that we don't see the same number of reviewers, we don't see the increase in reviewing and testing that needs to go along with that. And so we're in danger of causing a burnout for our maintainers, is my impression. A lot of people, I know some maintainers who've been doing it a long time and I don't know how they keep it up. It's a tough job, so yeah, more. Okay, that's good. Okay, so the comment is it'd be good if besides the stats for contributions, we also have the stats for maintainers. Who's maintaining what? And luckily we have the data on that because it's in the maintainers file, so we could extract that. One of the things, I was a little bit hesitant to put a table on here with some numbers because I don't wanna turn it into a numbers game, right? And different companies at different phases of their, not their life, but of their development or their work are doing different things, right? So Consolco had a really low number and I felt, and that's not because they haven't contributed a lot in the past, they just, I just happen to know they're working on AGL or stuff right now and it's not in Kernel. So you gotta be really careful looking at the numbers and making any kind of assessment about the value of a company. But I do think recognizing the maintainers and the contributions that people make is really good. And there are some, if you actually go in and look at the patches, you can kind of tell what the company is, what their specialty is, right? So Kolobora has been doing a ton of stuff with the Panfos driver and you can find examples, you kind of know who their customers are based on some of this stuff, but no. But I don't wanna turn it into a numbers game, but I think you're right. So we need more, other metrics that are good. Any other comments? Yeah, all right. Okay, so let me, right, so let me see if I can restate that. So in the Octo project, the Dunfell release and Kirkstones, they see contributions on those releases, presumably, because those are the ones that are being used and put into products and, but they don't see contributions at top of tree. And so a lot of stuff, as the top of tree needs, as they encounter issues top of tree, they're not getting fixed as fast as they'd like. And so a lot of stuff is going to an unassigned. That is the classic, I mean, that is a classic problem, right? So it embedded, a lot of times we get stuck not at top of tree, right? Our product designs, we have two, three-year-old kernels, two, three-year-old distributions maybe. And so that's where we're spending a lot of our effort. I don't have a good solution for that. What I refer to that as version gap and we've seen that throughout the industry for as long as I've been working in Linux. So I don't know if there's any ideas for curing that. Go ahead. Yeah, no, I agree. The comment was that it's a gap between consumer and contributor. So a lot of people are very comfortable being consumers, not as comfortable contributing. And so... I've seen that a lot. Right, yeah. So the observation up here is that SOC vendors and SOM vendors are often based on an LTS release. And so they're behind already. And... When you say, should we just start experimenting? Okay, yeah. So the observation is you do bring up on top of tree. Let me go real quick to... Well, I don't know. Let me... Any other comments? Are there... We've got about five minutes left. Are there any things that you see as particular obstacles to people participating? Is it as wide open as it should be in terms of people participating? That's a... Oh, comment back there. Okay, let me stop you there because I want to repeat. So the observation is that for a new person, particularly in an email setting, which email is a fairly popular way for projects to conduct their business, there's a lot of tribal knowledge, a lot of history. And so it's harder to come up to speed. So go ahead. Did you have anything else to... Okay. Okay. Well, that's a good observation. So one of the purposes of the wiki was to create a place where you could have this persistent knowledge, that you could jot down. But the trick is getting people interested enough. I mean, curl developers will rarely write a comment, let alone documentation, let alone go to some website and put some notes down. So, I mean, about the best thing that we've been able to do is capture a lot of the information that comes out at these events. So you can get people to kind of coalesce their knowledge and give it in a presentation. And then we record that and hopefully it's available for new people to look at. Are there any other ideas for encouraging? Yeah. Okay, so the question is, well, okay, a statement followed by a question, a statement that some of the reference boards that we have are a little bit old. They're supported by distros. Would it be possible to use a reference platform where it would be easier to collaborate on things like boot time? And if we're all on different boards, boot time and size in particular are very, very dependent on what it is you're trying to do, what your vertical stack is of software and also what your hardware is, right? Cause you're just gonna be tuning stuff like crazy. So I don't know. I'll be honest. If I'm working on a camera at Sony, I don't know how much time I'm gonna invest on some other board to focus on the boot up time. So that's a hard problem. That the paradox of embedded is really kind of a tragedy that we're in so many different places. We have this awesome resource that we've seen when we can collaborate. We can make great strides, but it's hard with the diversity that we have. But that's something to think about. Maybe if we chose some reference boards and tried to focus our efforts on a few targets, we might get some stuff out of it. Yeah. Okay, the observation is using an old reference board for measuring system size might make you depressed. Did you know that Linux version 0.11 ran in two meg? So yeah, okay. I've got to stop, but I'll repeat your question or your statement. And that is that we should expand our focus beyond just developers. We should look at management. And I would say end users. If there's a way to get end users involved in things like testing, my dream is that someone could take Fuego, which is my project, and run it on a product that they received, on a Sony TV and send reports to Sony. That would be awesome. So expanding to an even bigger ecosystem of users of our products as well. Okay, I apologize, we don't have more time. I'm gonna get with Garrett who asked to take notes and we'll look at some of these notes and we'll try to come up with, I'll probably see if I can send out a survey. I'm not sure where to send it to, given that we don't have a mailing list. But I could probably do the attendee list or something. Anyway, thank you very much for coming in your time.