 A little bit of content and a lot of interaction, because we're definitely interested in what everyone in the room thinks we should be doing, what we're doing good, what we're doing bad, and where we can be better. So we'll have a little bit of an update on the various stuff, and that's mostly going to be boring stuff like releases. And then we'll have an announcement of the Yachter Project Summit and some Q&A. So just a little bit of background for people that are new here. Open Embedded is the community project that was started. Armin wrote 2000. It might actually be a couple of years later than that, but it's very difficult to sort out when things actually began. I've tended to use a date of sometime a couple of years later when the Open Embedded development started, and you can find the first post is like Coon saying test. Open Embedded basically has no specified distro. You add your own distro conf that Baycore builds, QA is always sort of on and off. No real program management. We have a Wiki for documentation. Free membership. We have a TSC. You're on it. Joshua, Bruce, who are the other two? Yeah, there we go. So that's Paul and Tim Orling. And then there's an elected board that represents the community, and we represent the community to the Yachter Project as their technical partner. I am on the board. Tim Orling is new, and we'll mention him later. John Mason. I forget who the third board. I'm not going to go look in the IRC channel to figure it out. So the Yachter Project, it's been a long day, and it's the first time I've done the ball thing quite a while. The Yachter Project was founded in 2010 underneath the Concorde in Cambridge, England. It ships with a reference distro called Pocky, which is basically a collection of BitBake, OECore, and MetaPocky, and MetaYachtoBSP, I believe. It is basically the basis for the QA testing that was talked about by Alexandra in the talk earlier today. So much of this ball was covered in the last three sessions, by the way. There's some program management that goes on that organizes meetings on a regular basis, the documentation, does the bug triage. Membership is open to basically corporations, businesses, and if you're interested in membership, talk to me and I can get you steered to the right people. You must be an LF member to join the Yachter Project. Your company must be, and there's also additional fees on top of that. There's also a Yachter Project TSC where three members are elected by the advisory board from the Yachter Project, and two are elected by the larger community, which is open embedded. The governing board of the Yachter Project is made up of representatives of member companies. If that looks funny, if you're familiar with other Linux Foundation projects, this was the first Linux Foundation project, so it grew up differently, and I think a lot of things were learned along the way. OK, so Armand wrote these slides for me. And does anyone remember who the new member for the Yachter Project is in the past year? I think I was supposed to go and fill that out. OK, two releases in LTS as of this year, Dunfell and Kirkstone. There's a new online report for patch metrics, so that's showing you how many patches we're carrying and their status so that we know which ones we need to work on upstreaming, which ones are just not acceptable upstream and things like that. And it's a fun report to go and read. And we've fixed patchwork. It had had a bit of a rough spot and has been updated and replaced. Open embedded, we got around. We'd been on a bit of a bad spot because of the pandemic, and we had to reelect some board members. And Tim Orling is replacing Dennis Dementro. Oh, God, I hope he's not watching. Dennis or Dennis Denix, if you're familiar with him on IRC, as I always know him. If you want to become a member of the Open Embedded Community group and have a say in some of these goings, mail us a short biography to board.openembedded.org, and we'll batch those up and send them out for membership in the organization. Basically the members vote on the new members. And if you need to know why, it's a little bit of a cumbersome process. Just ask me offline. A lawyer wrote the rules, and I had to have it explained to me again. So we've been having virtual events, Yocto Project Summits, which are two to three day events with online talks. And the last one of those was at the end of May. And the next one will be November 29th through December 1st, similar format to the last one's presentations and one day of new training and talks, two days of talks. Yeah. Yeah. I have slides for everything. I just haven't read them in a, people don't let me up into stage very often. So after the Yocto Project Summit, we'll have a virtual developer meeting. So that will be online. We get pretty, it's basically scheduled to have good coverage in US and Europe because that's where we tend to get most of the people participating. It'll be open to everyone. And I've also got a note left over from the last developer meeting. Please take our survey to let us know what you use for layer setup and tooling. It's a one-question survey, takes almost no time. And there's a URL there that's going to be very hard to write down, but it will be on social media again after this for the next day or so. And that is open to anyone who wants to show up. So you don't really have to be a developer. It's basically showing people with an interest in the project to learn more and how to get involved. And it's a great way to get a lot of insight into why decisions are made the way they are. And now the favorite part of the BOF where I get to ask you to ask questions. Basically this is very free form. You can ask questions and either I will answer them or Mark Hattley will answer them or someone else in the room. Mark will be holding the microphone and helping me. I will be standing here on camera. Also follow us on social media. Joseph does a very good job of keeping our social media up to date. And you can find out about the exact URL for the survey because we'll send it out again after this. And we also are fairly active on LinkedIn and that can be a good way to stay in touch with everyone else in the community. So with that, does anyone have any questions? Yes, Sean. Thank you, Sean. Can you turn the microphone on? All right. There you go. Whoa. There. All right. So where can the community be found online? Like, say, IRC, what channels? So we're on a Libera chat. And the channels there will be Poundoe and Poundyachto. Yeah. Well, no. Thank you because usually it takes one or two questions. So I read it on Matrix and IRC and I think we just used the standard Libera chat gateway. Yeah. Okay. So you can't get to it? You can get to it from Matrix. Yes. It's just whatever the standard gateway is, however that works. I made it work for me once. All right. Because nobody else is asking questions. Another softball. What are the areas that we need the most help with on the project? People reading the bugs and fixing them. I mean, I think that's the easiest place to start doing things. And there's a list of beginner bugs that is published every week on the Yachto mailing list. And that's sort of a good place to get started. And certainly just hanging out in IRC channels and answering questions is also a good place to help out. In the last summit, one of the guys basically said he really appreciated it when people helped and it does look bad if people ask questions and don't get answers. So I will say that if you do go on IRC and you don't get an answer to your question, it just means people are busy. So try that again later. Yeah. A lot of the times you can ask it, two hours later ask it again, you get an actual answer versus silence. So don't feel discouraged by not getting an answer, just ask again. Sometimes you have to nag a bit, not too much. And sometimes, even if you know who might answer the question, sometimes just tag it in the channel so that they get a little notification that can help as well. If you don't know who, you might know who might have an answer to the question and ask them for help. If you're sort of like, I'll do that if I see a question that somebody I know can answer much better than I can. I'm beginning to think about asking questions of people I know in the audience. Touche. Can we have the mic on, please? Thank you. Yeah. Just leave the mic on. So what mailing lists would be good to join to get involved and how are patches submitted? Go to yachtoproject.org, go to the website, yachtoproject.org, and look for, you know, it's either a communication or mailing list. I'm not going to pull it up on my screen because what else will appear? Yeah, but I think the website will have a page that sort of gets you there. It'll jump you there. And I think they'll get more other answers to questions that Sean is answering there as well. Thank you. Hey there. So the Yachtoproject does not support UCLABC. And the reasoning that I was given was, you know, no one uses it, right? We don't want to support it, so we just kicked it out, right? So this year I had the fortune or maybe the misfortune of working on a camera project from a vendor, starts with real, ends with tech, right? They want nothing to do with Yachto. They basically give you an external tool chain and just off you go. So I said, hey, Yachto supports external tool chains, right? What's the problem? I can just export it. And then I realized, oh, OK, so the tool chain only supports UCLABC, Yachto doesn't support UCLABC, so now I'm at a crossroads. And in the end, I wasn't able to import the external tool chain and just stuck with what the vendor provided. So I guess the question is, is there any plans to maybe bring UCLABC back? Is some vendors are still using it? So I'll answer, and I think, Josh, you want to answer too, but we pretty much moved everything to muscle because nobody was using UCLABC anymore. We weren't getting any bug reports on it. It was broken for like a year and nobody filed a single bug report. And that was really our sign that it wasn't being used by anybody in the community or at least our part of the community. The other thing I would say is, if you might not need to use the vendor provided binary tool chain, that might be another option. You might be able to get around using the one that's provided by the project instead, and then you wouldn't have that problem. It just depends on your use case. I don't know if they're just giving you just the tool chain. If they're giving you like literally just GCC, you can probably just use the built-in one and not use their external one. But I don't know what else, I don't know specifically what you got going on. So that's how I've gotten the most traction is not using external tool chains. So that's a really good point. And we thought about that. And then it turns out it's a MIPS processor, but not a generic MIPS processor. It's their own homebrew thing that we won't share the code. So if you don't use their external tool chain, you're screwed. So any response to that? No. Yeah, as he said, I'm sorry. We've all hit that, I think, in different parts. So it's unfortunate. So I'll speak for AMD Xilinx very quickly. We have a similar problem with our MicroBlaze tool chain. For a long time, our tool chain, while we published the source code to it, was pretty much proprietary. And we've been working on changing that. But the reason why we're working to change that is customers came to us and said, we need you to change that. So while you might not be large enough as a customer, if you can convince other people to continue to go to them and say, this is unacceptable, you have to get your tool chain out, you have to release this stuff, they will eventually listen to you. And that's the best answer that I can give you. OK, I have a question. The mic? Yeah. My mic's working. Yeah. OK, so I have a question. We have a layer setup survey. And I'm just going to run a very impromptu version, and I don't remember all the picks. How many people here use combo layer to collect layers for building? How many people use repo? So that's about six. How many people use get submodules? 10 or 11? How many use CAS? Three. Are there any other ones that people are going to use? What else do people use? Wisk or something? Yeah, that's what I... How many people here take screenshots of the output of BitBake? And the one person that told me that did not raise his hand. OK. We can record that as kind of also a larger increase of the sample size. How many of you have taken the survey? How many of... How many here people have entered the survey already? Sorry. OK, so I got new... Richard says that U.C. Libc could come back as a layer if you needed it. What was that? He said it could come back as a layer, like a separate layer. U.C. Libc? Just not in OECore. OK. So, yeah, I get it on my watch, which is convenient. Yeah. Yeah, I told him I was going to cut the camera off. So... So the question is, how many people are new? Raise your hand. So how many people use the Active Project? It's pretty much almost unanimous, then, I guess, in the room. So the new, there was about three or four people. Yeah. So what are we doing well? So I think one of the big things is the documentation is incredible. Like, on first glance, it's really dense and thick, but once you kind of get past that, like, how do I use this type of thing? There are a lot of examples that are, like, practical, which I think is missing in a lot of technical documentation that makes it, like, clear what the intent and how to proceed with the piece. So A-plus for that. Like, it's one of the better ones that I've encountered. All right, thank you. Anyone else just want to say what we're doing well? Come on, we've got to be doing something well. Okay, what aren't we doing so well at? Does anyone have any comments there? Thank you, by the way. So we ran an experiment with two interns in our team. One intern trying to bring up OE, the other intern trying to bring up Buildroot. The experience wasn't that good. Yeah. Okay, just so you know, I actually have trouble hearing what people are saying because the speakers are blasting back that way. So I will need a little bit of help from Mark. What did he say? So he was saying that he has multiple interns and one of them was having a very big problem using it. So it's getting people started. Okay, so yeah, we don't make it easy to get started. All right. I mean, we know. And I don't have any great answers, unfortunately, but what I will say is I feel like we have a really good community and if you are having trouble, ask questions. Hi. This is taking out his own laptop. Yeah, I was going to say that we have our training materials free so you can probably pass that to your trainees. Right? Yeah, but this... Yeah. So one comment, and in fact, actually, I think Alexander made this statement or maybe it was Tomas in one of the presentations. So Buildroot is by definition targeting slightly different output. So it's a fixed root file system as opposed to a distro and Open Embedded in Yachto is giving you a fully functional distro that has a package manager and obviously there is a lot of overlap between those. But by its nature, yeah, there's a lot more complexity. You know, ask Buildroot to do the multi-config stuff that Mark did his talk on and it doesn't work. So unfortunately, with that level of functionality and that level of tunability comes a lot of complexity, which makes it hard. But we've been fighting this for years. We've been trying to improve this usability for years and we have talks, you know, going back years about how we have incrementally made it better. You go back to when the project was founded and it was even worse. Especially, and I wanted to recognize the documentation thing so Scott Riffenbark was a fantastic technical writer for the project. He's unfortunately passed away. So we actually, if you go to the page, you can find a memoriam form. But that's his legacy. So it's very good to hear. But to your point, yes, there's room for improvement. So that leads me to the next question for the audience that I'm curious what other pain points people have because you've got developers listening here in the room and they can take that back and maybe make it a priority. So here's your chance. Yeah, I guess my comment on when people say it's hard to get started, it's a lot more helpful for us to describe why it's hard to get started. There was a classic legend at one point that Tim Byrd had a lot of trouble with it. And eventually, we pinned him down in a bof and his problem was, his specific problem why he could never get a bill to go was his IT guys were so difficult to work with, he couldn't get the fetches through the firewall. And there's just nothing we can do about that other than possibly probe the firewall ahead of time and give you a message what needs to be done. So if there are little things that you see, please let us know what we can specifically do to make things better or what specific problems are rather than just say it's hard. That's my really best answer I have for that. I would like everyone in the room to please let us know what specific things they think would make things better. So the one feedback that I've been given in the past about getting started and why it's so difficult is because there's five or six ways to pretty much do everything within our environment. And what isn't clear is why are there five or six different ways to do what looks like the same thing? And it turns out because we have five or six variations that we need done and they were developed by five or six people. And if you see that type of stuff and you can't find any documentation that says why are there five different ways, that's great feedback because maybe we need to get rid of some of those five or six different ways or we need better documentation that says the preferred way is this. Now if you've got some extra. Yeah. Yeah. No, that's... I mean, it's a constant discussion how to present it and things like repo are very good for building a fixed configuration but they're terrible for doing development on a layer stack and maintaining the Git repos whereas sub modules lets me work with all my repos on the system that I'm developing. So there's two very different use cases there. I use sub modules because I work on all the layers at once. I've used repo and about became very angry with it. See, and I have the opposite experience. I use sub modules and it just makes me upset. So I use repo instead. I mean, I will tell you that sub modules has continuously improved. Oh, absolutely. I mean, they have much better commands to change branches and things. Yeah. No, they've all improved and honestly, some of the differences are we got used to doing it the way we were doing it. Yeah. And that's not always the best answer and that's why things like this, the mailing list, IRC, ask why something's done that way. That feedback is really important. Yeah. I mean, Joshua's talk on layer setup stuff is really good and that's on YouTube. Honestly, that's why these sessions are really valuable. Okay. Because if you've been in the community for a while, you don't experience the ramp up and we get, you know, like the veritable frog in the boiling pot of water. You throw one in and you'll jump right out. But if you slowly increase the temperature, it'll stay in there until it gets cooked. Well, we've been cooking in this project for a while and so getting the outside perspective, especially from people with fresh eyes coming in and saying, why is this this way? To what he was saying is extremely valuable. So don't be shy about doing that. We honestly appreciate it. So I'm going to realize a second. I just want to get the veterans takes. What are the tips, tricks, helpful hints for the new people or less experienced people with Yachto and OpenVid? Ask questions. Yeah. Ask questions. I just trained in some new people, I guess six months ago. And my first thing is don't try to do it all. You have a project you're trying to build, which is end to end everything else. If you try to do end to end, as soon as you start, you will fail because you've got to learn through the process. It doesn't mean it's going to take a year to learn it, but you have to go systematically through it. The other thing that I told people is unless you're starting with somebody else's starting point, don't try to bring in everything at once. Because you can very easily bring in 12 layers and then find you can't figure out how to make them work, but you can start with just the Yachto project or just the OE layer. Get that work. And then you can add your vendor layer. Get that work. And then you can add the next one. Get that working. And that's, it's somewhat obvious if you've been doing it a while, but if you haven't been doing it a while, it's you go, oh, I need Python. I need this. I need that. I need this. I need that. And suddenly you have 14 layers in there and you don't cooperate and you don't know why and you have no idea how to solve it. I was just going to say that I think being active is also very helpful. Like, you know, being an IRC, asking questions, if you don't get an answer, ask it again. Attending the technical meetings is very helpful as well. We usually have very good discussions on, I think, Tuesdays. Yeah. So I would recommend that. Also, Open Embedded has a happy hour once a month. I think it's the last week of the month. We have a calendar with all the dates on it and just show up and talk to people. You have questions, you know, random questions. It's a good place to meet people. And one of the other things is sort of like getting started. Like one of the guys earlier today, start with just getting it to boot through U-boot. Call that a success and then go on up. Especially every time you switch hardware. Even if it's to the same vendor's hardware. One of the things that I've told customers in the past and when I was working at it, we were competitors at one point. Good layer of discipline. Make sure that you are, because there's a tendency to throw together a recipe, cobble together from a bunch of different sources. And once it starts working, you don't pay any attention to it. And then don't know where to put it later on. So start out with a little bit of a layer concept of what layers you want to have for your stuff and what layers you are planning to bring in. That also leads to, you know, going and checking out, you know, what the layers are that you're trying to bring in. One of the nicest things about the Yachto project that it brings to the table is the Pocky Distro is a reference point, which is a really solid reference point. That's why it's out there and being built. OECore is part of that, you know, at its heart. So if you look at that layer discipline in terms of recipes, in terms of what layers you're going to bring in to Mark's point about, don't just bring in every layer under the sun. You'll take that incremental approach. You're going to get there maybe a little slower than you'd like, but you will get there. And what's interesting, and to your point about the learning curve, what I've also seen is that if you spend that time, you get the dividends at the back end because of all of the benefits you get from, you know, the cashing and everything else. Yeah, one more thing. Don't be afraid to make mistakes. The worst case is you can start over and use what you learned. But even today, and I'm quite experienced at this stuff, I'll build a layer and then six months from now, I'll go, you know what, I made a mistake. I need to refactor my layer. Don't feel bad about it. It happens. So another question I have for people in the audience. What hardware would you recommend for people who want to start out with the Octo project and a board? You know, what hardware is easy to work with? Yeah, Raspberry Pi or anything that MetaRock chip supports. Okay, so the Raspberry Pi layer, Meta Raspberry Pi and MetaRock chip. Yeah, that gives you quite a big selection. So I always tell people to start with QMU. You can get it working with QMU, then you can move to the hardware. Well, I specifically phrased that as hardware. Yeah, I know, but even... But thank you for mentioning that that should also be considered. And I work for a hardware vendor, and even I'm the one saying, start with QMU. I don't want to start with my own company's hardware, not because there's anything wrong with it, but because I know I can bring it up on QMU first. And then I can go back to the other people and go, hey, I can't get this working on the hardware. Can you help me fix it? And I know that it's not the software site. Also along that lines, in case you don't know, QMU supports like OpenGL now, so you can run your graphical applications and everything on there. So it's quite powerful. Yeah, we can run our entire software stack on QMU, which is really nice for testing. When we added RISC-5, a RISC-5 QMU machine, I sat down and built a GNU radio image and ran the P test in it just to say that I'd run GNU radio on RISC-5. So, you know, it's an excellent way of getting started. Mark, do you have a question for the... Oh, yeah. So I have maybe a question on best practices. So we end up delivering a whole bunch of different BSPs on a whole bunch of different hardware platforms to a whole bunch of different application teams. And so today, typically, we build these BSPs... Five minutes. Five minutes. We build these BSPs, hand them off to application teams, and then they write custom scripts to package this all up. And I've kind of been trying to convince people that, you know, instead of writing all of these custom one-off scripts, this is what Yachto can do for us. I'm kind of curious what are best practices that other people have used distributing final application images for multiple hardware platforms? You're probably getting better at that than me. I was going to say, I don't have a complete answer for you there, but working for an FPGA vendor, every board is different. So I had to come up with a mechanism that we developed a generic interface for each system on Chip, and then the generic interface has an extension mechanism, and that's how the end user actually explains what the machine is and everything else. But my mechanism-specific design links, I think it's a good model, and if you've got questions about how I did it, I'm happy to explore that with you. But it's not the only model, and that goes back to the multiple ways of doing it. Okay, Mark, can you let Joshua chime in and then I'll wrap it up real quick? We're on the countdown, Timer. Sorry, some of that I think comes back to that layer discipline, like really being conscious about how you define your layers. Like, we made a very conscious decision to make our distro layer independent from our internal BSP layer, which is independent from the upstream BSP layers, because we're doing consumer electronics. The upstream BSP layers are pulling in, and we can stack all these on top of each other, right? And like, they all share the same distro layer, which is awesome, and distro policies throughout everything we do. So that kind of thing really helps if you want to explain it more offline, if you want another example. Okay, so it's time to wrap this up, because my eyes aren't quite good enough to see if it's two or three minutes. Thank you all for coming. Please take a look at yorktoproject.org and open embedded.org for ways to stay in touch in the future. We love to hear feedback from people all the time. Join our IRC channels, ask questions, help people answer them. Take a look at the beginner bug list that is prepared by one of the technical teams. And I look forward to seeing you at the next conference in Dublin, or I think it's Vancouver. Yes. Okay, thank you all for coming.