 So thank you all for getting out of bed so early for this talk My name is Philip Ballester and we're going to talk about the Yachto project and its direction hopefully over the next five years And this is the introduction slide that I'm not going to talk about and just a little bit about me I've been using Linux since about 95 a which I wrote the version because I can't remember the year and I ended up in grad school in the 2000s and ended up using open embedded to build a Linux distribution for no map one and Based my master's work on that Along the way, I got involved with the GNU radio project. So I don't do a lot of GNU radio on arms work And I was in Cambridge in 2010 when we found out about the Yachto project and its creation Just a little bit of my industry experience. It's been a long strange trip started in brushless DC motors contract assembly steel mill automation is where I started learning Unix graduate school and Mostly I make money off of people who want embedded Linux for software-defined radios so Why did the Yachto project come about in 2010 embedded Linux was really taking off and open embedded existed to build embedded Linux distributions at the time and So the use is skyrocketing and a lot of the people that did the early work on it We're graduating from school. I mean, it's basically very much a hobby project at the time for people who just wanted to run Linux on random devices Commercial users had taken note. They were hiring anyone who had open embedded experience There was no clear development direction or no clear development processes And it was just creating a lot of conflicts the hobbyists would break the build over the weekend The commercial guys would break the bill during the week and everyone was angry at each other So this Created the conditions to start the Yachto project and the Yachto project was the very first Linux Foundation collaborative project So it was very new and nobody really knew how to do a collaborative project at that time And I think we've learned a lot of lessons over the years the founding members included eight companies and the open embedded organization and Okay, so I talked about the collaborative project a little bit about what the Yachto project is practically these days It's corporate. It's a basically a company membership based organization You have to be a Linux Foundation member there are three levels silver gold and platinum It's now grown to 30 members and open embedded and the 30 number is very dynamic at the moment there's growth is accelerating and You're the company members you support a lot of different operations in the project an early one was documentation and One of the things I've heard consistently over the years is one of the best things the Yachto project has done for embedded Linux is Having good documentation and making it much easier. I mean it's still hard But making it much much easier to get your job done It's supporting patch integration and testing for open embedded core and metaparchy and meta Yachto BSP And that's a very difficult job because there are a lot of different possible build configurations And it's very easy to submit a patch that works for your Machine distro case, but doesn't work for others the classic patch failure at the moment as it fails to build for muscle because only a handful of people are testing for muscle, but What we're doing with the extensive testing is keeping it working for everyone Behind this testing, there's a lot of infrastructure the auto builder as you can imagine doing builds For a lot of different configurations is very stressful. I'm sure we've all seen Ross's talk on making our builds go faster Now imagine testing You know how many machine? How big is the test matrix? Yeah, so there's 50 jobs and that's you know per bundle of patches So you want to go really fast to get the patch keep sucking in the patches as fast as they're generated There's a lot of work done bug tracking auto builder failures and other failures There's testing the pocky reference distro on a variety of actual hardware. These are the reference platforms And there's a lot of work for advocacy and public outreach, you know, they're supporting the Octo project stand supporting developer days and other acts activities to educate and Introduce the Octo project to new users It's also managing contractors some of the Member dues are going to pay people to look at help manage the auto builder and fix problems and keep bugs up to date and Developing new funding streams. So, you know, member dues can't do it all So we're always looking for new creative ways to fund embedded, you know core embedded Linux work This slide I put together very quickly I realized that many of the stands here had some form of Yachto to project dependency and I had a pretty low bar If you understood the word Yachto project and would admit to have used it for something I would count you But 18 of the 24 stands and the ones that are not really using it are a lot of the hardcore Zephyr only people so We started a process at the advisory board level to answer the question. What should the project look like in five years? Historically a lot of the work has been people scratching their own itch itches contributing code as they can and It's not been very focused We have a lot of ideas. We have more ideas than we have people to implement them So part of this is, you know, hey, we have these great ideas or we have these great tools that are sort of bit rotting I mean, I hate to use that word, but it's it's accurate and we'll talk about some of those in a bit and We're trying to get improve the project's sustainability as I talked about Companies have been hiring anyone who has any open embedded experience or Yachto experience and The problem is that's leaving very few cycles for core work and improving the overall ecosystem and We're always talking about best practices. We need to document them you know, we regularly see talks on BSP best practices and We need to improve the project reveal resilience, you know the bus factor We need more people who can work in core and are familiar with stuff And we're always looking for new members for the Yachto project It basically gives companies a better voice into where we're going and it gives us more money to help fund work like this So After creating the plan, which was, you know, several months of meeting and learning about how to Score ideas and things is very educational process for me at least Estimates were developed, you know for implementing some of the ideas and Of course, you know, we have these estimates and we have no one to put on the tasks But at least we've bounded the problem and we can talk about it now Yes, you're all working I know We're already funding Richard Purdy We're paying developers to test patches and report issues and we're paying for documentation work and that's Thank you, Michael And so after looking at the five-year plan the conclusion is we need more members to help fund this work and improve the entire ecosystem so Now we're going to move into Some of the details that we identified in the five-year plan So some of you may remember patch test was running for a while and it would basically you would email a patch And I know that the email to a list workflow is questioned by some but this is what it is So it sends patches via email Patch patchwork will manage the patches coming in so we don't lose track of them And patch test had been implemented on top of patchwork And it would run some basic tests against a patch and then email or reply to the list serve And that was great up until the point that we chose the wrong branch of patchwork And eventually realized that we were going to have to change patchwork branches to manage patches and that broke patch test By then the patch test maintainer had moved on to greener pastures And so no there was no more patch test, which was very unfortunate So the plans are to update patch tests to work with the new patchwork There's also the idea of running the basic tests locally So that you don't have to mail the patch and have it patch test report that you made a very simple mistake And as we get more experience with patch tests, we can change the patches to help catch common errors But the end goal is just to make it easier for people to submit patches and Have sort of more prompt feedback and give people more confidence to submit patches So I how many of you remember toaster? How many of you liked toaster? All right Yes, okay So toaster has been around since almost the beginning of the yachto project And there's always been a lot of effort to improve usability and It would basically collect it was built by Some actual user interface designers which I mean I was in one of the interviews for it and I admit that I am they type bit bake. Why do I need any other tools? But a graphical interface does help new users and we'll talk more about that in another item and If you type bit bake core image soda, this seems simple But the question is what is inside and what I found to be the most useful feature of toaster Was to turn it on after I've done my build and then use it to follow my dependency trees around Coming we've all made dot files and then realized we can't render the dot file, but we just read the code to work out What dependencies are there? But toaster gave us a tool that let us look at the entire build and examine why things happened So part of it is there is toaster is Sort of working today. It needs a maintainer We've automated tests are one of the things that help reduce the burden on maintainers. So we're trying to improve testing We'd like to get more people to actually use toaster And I think we can do that by having relevant features that help people And we want to work with users to figure out what they really need and make it a useful tool To support the user base This is not my favorite slide and I might wave at ross a little bit So building everything from source does take time and this is a source based Linux built Linux distribution build But many of you are building the same things and we have a lot of Support in the project to minimize rebuilds and the idea is we can share some of the build artifacts so that You can point at a server that has a build artifact that's already built And since we're very careful about rebuilding and hashes and all that We should be able to have some public data to improve the workflow So there's public shared state on the auto builder today that you can point at. It's not Well set up for bulk use So we'd like to get a real content delivery network set up for it And there's also as always This needs a few extra bits of code to make it work really well so we've got project tooling and These are tasks designed to make things easier You things like recipe tool and dev tool. We've got some fundamental tools like sudo That help us build file systems not as root And once again the theme here is that people work on these and then they get moved to other projects and it Is on live support? I mean, I've used recipe tool quite a bit just if I have to build a recipe really quick I mean, I ran it against. Um, oh, what is that? It's one of the ai things pie torch And actually managed to get 90 of a pie torch Recipe working. I mean it had problems that detects licenses poorly Which led to a lot of editing, but it did a pretty good job But you know recipe tool doesn't understand some of the newer build systems I think later python. It doesn't understand. So there's work that needs to be done so Okay, so this is the plans for project tooling. So we've got a bug backlog for things like recipe tool and dev tool We've got modern languages rust and go that recipe tool does not really understand and they have their own packaging things And there's always people always have new ideas for things that can make the projects better and easier to use um So in the beginning Open embedded classic, you know circa 2009 was what we would call one layer with all the recipes in it And as you can imagine that was a nightmare. Um, you would have a lot of multiple versions of recipes because of dependency issues and I mean it was just multiple thousands and it was untestable And was getting unsupportable So a lot of the maintenance challenges with one giant layer led to the conditions that created the yachto project So open embedded core made a testable base system, which has been very good for everyone in this room But there were still a lot of recipes that didn't make it into core and they got moved into meta open embedded Which is divided amongst several things like meta python meta oe meta networking But this is basically maintained on a As people find problems they fix them So there's a lot of recipes that may not work very well don't get updated on a timely fashion and the maintainers are Basically like i'll bump fftw when I see it and I got annoyed with fftw because it broke on a set of current Compilers that were in dunfield. So I added the patch test, but there's no organized effort It's people doing what they need to do to get their job done But it's still very heavily used because there's a lot of useful stuff in it So the plans are we'd like to add automated cve check-ins similar to what we do in open embedded core We'd like to run the automated automatic upgrade helper against it so that we can keep things up to date without depending on people noticing that changes were made Work on adding more p tests Mirror the sources for the lts releases. I there's mirroring done, but it's very informally hand managed It's basically like you notice something's not in the mirror and you start emailing And it gets fixed Identify obsolete recipes and remove them. I mean they're projects that are just unsupported and should go away Which may come back if people read re-energize these projects And as always improve reproducible builds Security is a big Thing or it should be a big thing And it's more than just running tools And we need processes in place To handle security problems when we're notified of security problems. I mean we Really need a formal security process We've been working on supporting some of the industry security Initiatives like s bombs and there was a very good talk at fosdom by joshua watt on that But there really no dedicated people to do security It's basically done on a semi volunteer basis But there's just no like point of contact for security issues So I mean we need to identify and document the current processes And we need more security infrastructure. I mean most people will have you know security teams That whose job is security Which is creating staff a security team And the s bomb generation needs to support s pdx 3.0 Another really interesting one So I wrote all the cool people use ide's I'm not cool and You know some of the earlier initiatives in this area. There was a eclipse plug-in that was done by a guy from bug labs That would help you write recipes And that being a clip so maybe didn't get the attention it needed But the idea is if we can have a structured workflow in an ide we can make it easier for people to get started We can integrate with the build environment And when you're talking about ide's people immediately start to get confused Excuse me So we want to support both the recipe developers the people writing bit bake files And help them do their job and there's also the opportunity to create tooling for application developers to hide the complexity of Debugging applications that are built with the octo project so The idea is you can have a vs code Terminal that knows how to do remote debugging with your target hardware or kemew And knows how to integrate with the tool chains So there's just a lot of really interesting work that we can do with vs code and You know there's some Prototypes floating around that are very interesting So what we'd like to do is put in place some people to Complete support for writing recipes, you know, obviously the prototypes floating around we'd like to get organized Work on them so that we can maintain them Once we have some of these plugins written we need to worry about having testing in place so that We can find out if they're regressions We need to work on stks for application developers integrated with vs code so that I mean application developers are notorious for not wanting to run bit bake But if we can Set it up so they can start vs code and just do their job. That would be a great leap forward And we want to do testing and debugging on kemew and target hardware We know there are a lot of existing tools and we just need to get them all organized under vs code or Eclipse And we need to talk to the user base to see make sure our efforts are are really benefiting people So that's it's really important you know to talk to people about what you really need and As long as i've been around we've been talking about binary distributions and how to get More devices so that people can prototype quicker You know we can build multiple images from the same bill and we have shared state which is It's not as convenient as a standard package format, but it is It's own sort of package format is is kind of the thing here And we know we can do things like this because once upon a time a very long time ago There was actually a tool that would build images you would type the image name And the machine and it would spit out an image that may or may not work on your hardware But it was a very interesting thing at the time. So this is sort of an old idea that is new again So what we want is a system like that that will create deployable working images that really help people prototype hard prototype faster This is as always much harder than it looks There's questions of what features should be included. How do you know what machines should you spit out containers? There's just a lot of interesting things How to actually do this in practical And my favorite one So layer setup I think the the saying is if you get three developers in one room They will have at least six ways of doing layer setup And of course open embedded classics solve this very neatly by having all the layers in one repository This may not have been a good thing So today a useful system will have open embedded core and a bsp that gets you a very basic linux distribution Typical systems are going to add a few more layers And then there today there are many existing systems solutions repo was state of the art 10 years ago Combo layer is what parki uses get sub modules is what i use And alex canavan has put in a bitpake feature and there's cas and i'm sure that several of you have ones that are not mentioned here So as I said three developers six plus solutions So the notes on this basically said that this is a challenging people problem because Everyone has their own idea what this should do Things like cas look to me like they're very good for doing continuous integration because you can basically describe the build and the layers I like sub modules because if I need to edit a recipe and Send the commit out I can very easily do that and not lose my state of my build You've got to track the layer revisions and the build configuration So you've got to keep track of the con files used to set the build up Um asked me about the time I had Image features defined in a local.com that led to some embarrassing revisions And once again, this is just a people problem. It's figuring out what people need what they can work with what their needs are So How can we help get this done? I mean the five-year plan is basically nothing but a plan and Early on we had to admit that there we didn't have any way to staff the plan or convince people to work on it So If you're with a company, please join the yachto project You know, it's Just it really helps to have members it gets you more insight into the goings on of the project I mean when i'm writing this talk I'm thinking all this material is very familiar with to me Because I see what goes on inside the project and it helps just to be able to You know attend meetings have the opportunities to work on stuff like this You can also we would love companies to dedicate resources to support the project is you know, don't Tell the guy he can spend, you know one day a week on it, you know not His weekend on it, which is what happens in many cases is people are working extra hours to help out And there's been an rfq published. We have some funding for tasks in the five-year plan So look at that rfq and if you're in a position to bid on them, please do So individuals Talk to your employer about joining the yachto project It's probably easier at the smaller companies I know when I worked at ge having this conversation with my manager would not go well Because he would have to talk to his manager who he didn't want to talk to But you know talk to them about joining the yachto project or talk to one of us and ask us to help you We'd love to do that If you have extra cycles at work, you know talk to your employer about upstreaming some of your stuff If you have ideas, you know share them at a yachto project developer day, you know Let us know what you're working on what you have prototypes and maybe you can get some help from other people with it And if you need some money look at the rfq and bid on things You know, it's open to pretty much anyone That can be paid So if you have questions I wrote this slide not knowing how long this talk would take so if I ran out of time I was going to refer you to the boff And I've been told that many people don't know what a boff is anymore So there's a phrase birds of a feather flock together So basically if you have an interest in yachto project and open embedded show up at the boff and just ask questions You know, we've got a couple of slides On things but it's basically a chance for you to talk about things that you've done your problems ask questions Okay So does anyone have any questions Question from the remote audience about security. Can yachto provide Harden configuration for system d services Take the microphone to ross Uh, it should be able to uh, I haven't tried it. So this is a hypothetical. Yes All right. Thank you Yeah, the hardening features are mostly up to configuring your system d service file correctly So it should just work as long as you have the necessary libraries and able to like like lipsec com for example So this is something that we probably should document better is how to build hardened Yeah, but a lot of people doing this don't know what normal system d stuff is You remember what I said about documentation being very popular Any more questions come on you have to have questions. There we go Thank you for the presentation. I think that was very useful. Um, just on the kind of release cadence Looking to this this five years At the moment we are releasing on a six month cadence with the fourth being an lts. Do you see any Plans to alter that at all or is that going to continue on as you say for the next five years? All right, the release cadence. I mean there hasn't been any talk of changing it. So it'll just continue As it does And I can't have we said anything about lts's? publicly So the lts will continue as is but be extended to four years per lts Great. So that is one change and please When you get to the end of the four years update your stuff I love the nervous laughter there Okay, more questions Come on. We have eight minutes to fill Okay. Oh, there we go And there was another hand there I saw What about lowering the baffle membership? Um, as I'm self-employed and I have several projects, uh, customer projects which are using yokto and doing all the Convincing stuff you told about is pretty hard. But um, what about the broncy membership which allows to to step in earlier? Okay, so the question here is we need a lower level of membership that lets people Get in a little bit easier um, this is a fascinating question and Being sort of the community rep on the advisory board. I have made this argument several times And I haven't been able to make that happen um But thank you very much for asking that question so with regards to membership the silver membership is based on company size And so The bigger the company the more you're going to pay right for so for smaller companies it's About five k So $5,000 for a small company up to I forget the exact number of employees Right. Yeah. Okay. So individual individual contractors or That is something that we Hey, we do but yeah, it's work in progress that the biggest barrier we have is the requirements for um Linux foundation membership. Yeah So that that is the the biggest barrier to entry for a lot of people Especially uh individuals, etc Um, but it's an ongoing discussion that we are having Was there another question? You've mentioned briefly that not everyone likes submitting patches to mailing lists. Yes. What about ideas for um making um If creating and supporting different contribution methods like uh poor requests sending patches without mailing lists Can I I have I get I get young developers. Yes. I do not know how mailing lists work I'm going to refer you to the ball for this part of the discussion Yeah, it's probably best to take this to the ball, but this is an area of ongoing debates and We don't want to raise the sending patches Yeah, it It's a challenge I was just going to mention we uh, do realize that there are some confusing paths to take in the contribution guidelines and the documentation and Michael and the team are actively working on resolving this to just make it a bit clearer for contributions So hopefully that's a starting point we can all agree on and see what the future has in store All right You mentioned an rfq document of the future Evolution, uh, but I don't know how to Were to find it. It was sent to the mailing list. Um, the octo mailing list. Yeah All of them And and it's right on the front page of Yachter project.org All right, so we take as an action that we need to make it a bit more obvious on the yachter project front page And get it out of the scrolly bit And there's a question in the back Now we need a question in the front to make joseph run back Philip Maybe you could give us a background over why crops exists and what's the plan for crops? Um I'm not the best person to do that. There was a talk earlier in the week on monday Tim Can you answer that please? Um Can you give us some background on crops and why it exists? Is that right plans moving forward? Okay, so yeah Crops originally was created back in the days when everyone wanted to use, you know Eclipse on windows or they wanted to use their mac laptop to build the octo project So that was a very very original part of it The only way to solve that was with containers and so that's where we started It morphed into something completely different because we were no longer trying to just support an ide with sdks We were actually now basically trying to build yachter project with a completely reproducible Base right you throw away worrying about the host Uh requirements completely you never think about it ever again, right? Okay or almost So that's what it turned into and what we're doing now is we realize that The base containers have grown a little bit long in the tooth What they have in them is not exactly what we need anymore and it's not It's definitely not what everyone needs for everything So the way to extend those containers has turned out to be completely non-obvious to everyone even though it's not That hard to do right? It's just like ross said it's just a system d thing. It's just a container thing So We just gave a talk on the crops generator. So this is brand new, but it's using kconfig to create Containers for you and the entire plan here the entire reason for doing that Is to support vendors who need to put their sdk and everything into it and be able to ship either a container Or a very easy way to build your own new container And that's that's where we're going Thanks tim So we got a couple minutes any last minute questions one minute even one more question All right, thanks everyone for coming I'm hopefully this was useful