 Okay, we can get started. Welcome to the Yachter Project and Open Embedded BOF. My name is Armin Custer and I work for Monivista Software. I've been working on the Yachter project for about six, seven years. I'm on the technical steering committee for the Yachter project and I'm the stable branch maintainer for Paki and Meta Open Embedded. Hi, everyone. My name is Dengla Deshain. I work for Linaro. I'm the committee manager for the Yachter project. So we definitely want to thank you all for being here today. I've been involved with the project for a couple of years as well and we do this BOF at every of this ELC conference. So I mean, this is usually the time when we really want you to ask questions to the developers and the Yachter project people. So today, we are going to start with a couple of information that we want to share about project updates and status updates and then hopefully it will be quite quick and we'll open up for questions. So we have a few people from the project in the rooms. Hopefully we can answer the questions and if we cannot answer the questions, we'll get you the answers as soon as we can. A couple of project updates. So this is a fun one. I don't know if you've seen this slide. I mean, that was actually one of the recent Linux Foundation events that was presented by Jim from the Linux Foundation. The Yachter project has basically put Linux on Mars. I know if you missed that news, if you want to learn more about that, I thought it was a fun fact to share with everyone. There is a small module that actually runs Linux that has been used for the Mars Insight project last year. So if you want to learn more, I mean, you can ask Google, you will find some interesting data. That's a fun fact about the project. Since we've done the last both, it was an ELCE last October, the project and the developers have made a couple of releases and we are actually working on the next release for October. I just wanted to give a couple of numbers and all numbers about that shows, I mean, how much changes and what we are actually doing. I mean, how many people get involved in the release and how many changes. So it's, I mean, it's definitely a significant amount of change for the size of the project and number of the developers. We've done a couple of stable releases and as I said, we are marching toward the 3.0 release and we are in the middle of the cycle for now. Yeah, so you noticed that we are actually switching, we've decided to switch from 2.7 to 3.0. The reason is actually on the next slide. We are very happy about the state of the project for different reasons. Many changes happened last year. So they are technical, that's a lot of the optimization and the recent changes are you're going to learn for the next release and so we are very happy with the state of the release and so happy that we actually want to change and make like a much bigger number. So that's, we've completely refreshed the hardware infrastructure that we use. You probably know that building all the York 20 open embedded recipes takes a significant amount of resources. So we refresh the infrastructure. We've completely redesigned the way we also do the testing. So we have now pre-testing before each of the merge. We run on all the testing on QMU and emulation on hardware like X86 and ARM, which is also new. So this is a significant release that is going to happen in the next few months. The next change that also happened this last year, which is not a technical change, but a very big change for the project, is we've ratified a new government's agreement and we have formalized the creation of the technical steering committee. So it's a big change for the project, the way it's organized. The TSC is made of five people. Three are actually nominated and elected by the York two members, board of members and two are coming from the open embedded side of things. So you have the name here. And overall, the responsibility of the TSC is to everything that relates to the Yachto project, all the gate trees, any kind of issues with maintenance. The infrastructure, the testing, the release process, the compatibility program, everything that deals with the Yachto project and that this is now under the responsibility of the TSC. The goal is not to have the TSC decide and do everything, but when there are issues or when decision needs to be made and that the committee cannot reach to an agreement, we will actually go and use the TSC for that. There's a wiki as well, if you want more details. Okay, so one change which we are going to do soon, we've started to mention that on a couple of mailing lists, but the change is significant enough that we wanted to use the opportunity today to have like lots of, I mean, quite a bit of people in the room. We are going to change our mailing lists. And the mailing list is basically the heart of, I mean, everything that we do as an open source community so it's a significant change. Many projects have started to use groups.io and switching away from Mailman or new projects are actually using groups.io without never being able, never used Mailman before. So this is, we are planning to make the change and Michael is in the room here. The week of, next, the week of September 1st, the weekend of September 1st, which is a public holiday in the US. We hope that not so many emails will be sent that day. And we will send instructions. If you are on a mailing list, we have 3000 people actually on our mailing list. You will get all the instructions, all the account and everything. I just wanted to mention that this is going to happen. We will be on the hook to make sure that it does not have too much impact on the workflow on anything. And for now, we are only going to do that for your YoctoProject.org mailing list. If everything happens and everything goes well, we will also plan the same migration for open embedded mailing list, which is where most of the patch and most of the discussion happen. But just that you know, one small technical issue and that we have to highlight is that the name of the mailing list will change. We had to deal with some DNS issues or limitations. And so we are actually moving every list to YoctoProject.org to list at dot YoctoProject.org. Again, if you have any questions, any concerns, if you hate Groups.io or if you love Groups.io and you want to talk about that, come to me or come to Michael in the room and we can discuss and chat about that. Something else that I wanted to mention also that we started three months ago. It's quite an interesting story. Someone from the community stepped up and came to us to the advocacy mailing list of the project. And he wanted to contribute something and it's time to the project. So Joseph is based in Germany. I said I want to basically start doing some live streaming sessions of me and showing what we can do and how we can actually use and start with the YoctoProject. So it's meant to be mostly for people who start with the YoctoProject, like an introduction sessions and how do I make my first build and so on. So once every month, and we've done that for five times I think already, is coming and basically does like a simple thing, like build my first image, build my first recipe, make my first layers and so on. So it's very open about what you will be talking next. The next session is on September 10. If you have not watched yet, I encourage that you just have a look. If you have any idea about what you would like Joseph to talk about, let us know. We can give the idea. If you want to help Joseph and make your own sessions, that's also something we will welcome. It's a very interesting story because it came out of the blue from the community. There are many ways you can actually contribute and help the project. I mean, it's not just sending patch or sending money. We also take patch and money, but just doing anything like that is actually very welcome and we are very happy to see that. All the videos are also uploaded on the YouTube channel if you want to look at the past video. Next thing, which is also like a big thing which happens this year, we are going to do the first ever Yocto Project Summit. So if you're here and if you know the Yocto project, you're probably familiar with the Yocto Dev Day. We've done that many, many, many times. We used to have like one day where some, we gave like trainings about the Yocto project and open embedded. So realize that the trainings were sessions about new things or topics about the Yocto projects. What we really wanted to do is slightly different. We want to submit like a conference about two, so that we get a chance to have the developers and the users and everybody. I mean, that I is actually involved with the project or just using the project to come together for like two days. The first day will be mostly presentations and talks and we are actually opening for CFP right now. So there are the instructions if you want to submit your own talk and we will actually decide the agenda, I think sometimes by mid of September. We definitely encourage you to come and tell us about what you do. I mean, so you probably think you are not doing something very interesting with Yocto or maybe you think that everybody else is doing the same thing, but we really want to hear what you guys are doing with the Yocto project. Whether you make a product or whether you will have some contribution. I mean, we are very open about hearing from you. So it will be the two days right after ELCE. It's in France, in Lyon, and we'll have some reception evening, social event the first evening. And yeah, you are very much welcome to join. And yeah, so that's it in terms of, I mean, a couple of updates we wanted to share. So usually this both is really up to you. I mean, to come and ask questions. So I'll go down and I'll share the mic and we'll try to get the answers. So we have a couple of people who can actually answer, hopefully all the questions. If you don't have any questions, that not going to be a really good buff. So I really encourage that you come with questions. So who wants to start with the first question? Yeah, you might need one, Mike. What are the barriers to an LTS version of Yocto? Why do you start with the tough question? You want to? It's a couple of things. One that keeps stopping us initially has been QA, keeping and maintaining the quality that we have for previous release. The other piece is mostly political. How do you decide an LTS for the Yocto project? Who's going to make that decision? And obviously the maintenance is, who's going to maintain it and what they maintain? Do you continue doing what the stable branch maintainer does or do you have something different? So it comes up every year or so in the meetings. Yeah, yeah. So like what are you thinking of? So as Armin said, I mean this is something which is, I mean one of the biggest requests we hear from everywhere. So this is an open source project. So I mean the project is good at actually maintaining the master branch and adding new features. And I mean there is no doubt. The maintenance right now, I mean the options is to actually engage with someone. I mean I actually will do that for you. About the project being able to actually itself do some LTS. I mean some other, I mean like they, I mean Ubuntu or all that. I mean this kind of model where one specific release would be maintained for maybe five years. It boils down to actually resources. And the problem I mean at this moment, we don't have resources. So if the open source community by itself and for free is not going to do that, which is obvious. The other option would be to actually collectively, I mean through the Yocto project membership for example. I mean we would add more resources to actually do that. So that would, I mean this is something we could discuss. If there is enough members, enough companies interested in joining the project and say I mean this is what we want. I mean the Yocto project is based on membership. So every people join, people vote, every decision is voted and so on. So if there is enough resources to do that and if the membership agree to do that, that can happen. If it doesn't happen collectively for free by the open source community, it can happen in the membership. And if it doesn't happen in the membership, then you have to do it yourself. That's it. So and the discussion we've had is yes, some people are interested, some people could join if we were doing LTS, but right now I mean this is not materializing to something which exists. But that's something we discuss very often. Yeah, so we work in the same company and we also have the same requirement for like seven years after the last sale. And we currently do maintain LTS for Krogot and THAT and we plan to maintain them for at least seven years. So and we also get approval to share our branches somewhere on GitHub. So we plan to do this. If somebody is interested, they can just use our branches. But we obviously don't do the full testing cycles Yocto does, but we do use them in production. So they are get tested in the main product QA cycles. This is good, really good. The only issue is it gets tested for your very specific use case and product, which is not the same thing as being tested and certified as a release by Yocto project. But it's the first step. So again, I mean the only answer, which is unfortunate, but the answer is definitely, I mean there is resource involved. So if there is enough people interested and willing to participate and contribute resources, yes, I mean we are very open to do that. Especially with what I mentioned earlier and all the improvement in the testing. A lot of the testing has actually become like automated. So I mean if we want to run more release, what we need is more computers to actually do the testing. I mean to some extent. So yeah, there is a scalable process, but we still, I mean we will still end up into missing some of the resources. Yeah, of course, but we are still, we are doing better with automation. We need, we can tell you, I mean Michael, we can tell you what we have today, so. So the tests right now are running on two generations old, rather high end Intel CPUs. They have at least two terabytes of storage and 128 gigabytes of RAM. And we use an NFS backing for part of it. So we require 10 gigabit network on these machines. Yeah, yeah. These are fairly similar machines to what you're using. I think we're at 96 cores, but very close. I just want to go back a little bit with the LTS discussion. And the question I always have, and I heard this discussion many, many times before is, what do people actually backport for more maintenance in LTS? And when do they stop backporting things? The question is, you know, when does the effort to do the backporting by far eclipse the effort to keep on a rolling, progressing move forward? And, well, I don't have the perfect answer for that. Well, there's probably no perfect answer for that, but I would like to understand from people who are actually doing backporting. And I did some of that for some things, for some CPUs and things. When is just not worth the effort anymore? And the other experience that I made is, you can push moving on to a new release or so for that much time, but then the effort that you have to put in is a big, big step forward to move forward. And for me, it always looked like it's easier to do small incremental continuous integration, keeping up the date with the new releases rather than doing the LTS in the backport effort. Anyone who wants to share some experience about that LTS or any reason why people cannot upgrade? I mean, that could also be a valid reason why people cannot keep up with the anybody wants. So in our case, we do regular updates on our main development branch, but we have stable branches that customer like to stick on them and they expect only some minor bug fixes and security updates. So for that one, we are using kind of LTS model. So but the main, like it's too risky to update them because the stack, the Yocto is not only the part, like it's only small part of the whole software stack. So if you update Yocto, it has a ripple effect through the whole software stack. So it makes it not that stable branch anymore. Yeah, security and bug fixes. So I think this is a good feedback. We can always take it to the AB, but I think people have discussed all the pros and cons. And I think LTS, as with time, it has maintenance cost as increases. And I think so there are policies, like what policies you want to institute? Do you want to do package upgrades? Are you doing new feature development that needs newer libraries? Now, do you want to build, you know, backport new libraries or not? What's your policy for LTS? So and that defines the cost structure to it. And it's not something small. And for an open source project like this, you know, this is a big step. And we are certainly interested, as Nico said, but again, you know, we are limited by our resources. So the TSC for the decision about the policy, but the AB for the resourcing on the thing. Yeah. So any more questions on LTS? I mean, that seems to be like the topic. I think it was the same thing as the last part for me, LTS came a lot. So maybe we should do something about it. Anyone? As I said, if you don't have any questions, that's not an issue. Any experience? So this is going to be the most important question of the day. How do you guys come up with the names for the code names for the releases? Just in general, what's the primary reason behind the code names? So the algorithm is simple. Richard made the decision. So one day we just learned from Richard what the next release name is going to be. That's it. And so, yeah, that's how it came to us. Okay. I think we have stories about the very first names. And then, I mean, we... Yeah. So these are game characters. And earlier, they didn't follow any convention, but now you will see that they are in chronological order. Like, you know, they're alphabetical order, at least. So, yeah. So I think, but they are totally random. There is no logic behind them, as such. Yeah, I think the latest alphabetical series has been from Total Annihilation, and Zeus is probably the last one in that series. So there'll be a new video game. I know there are several people that wanted to ask about CVEs. Are they in the audience? So I think it comes to the same LTS conversation. I was involved in Yachto over a year now, and one of the first impressions was it's great. It's used by everybody. But why do we have so many CVEs in these stable products? It just doesn't make sense. Then I realized that every single company is doing their own CVE-backport business. Nobody's actually giving back maybe a few companies, I think Intel is doing a great job, and I think I also saw Garmin as well. So I think the, as a user, it gives bad impression that, you know, we have all these vulnerabilities, nobody's addressing them. So what do we feel about this? So the question is, who wants to answer this one? So yeah, contribute them, right? Nobody's saying no. Essentially, we have what we have, and I think that's a good feedback. But I think, as you can see, that most of projects, you know, who is doing the security updates, you know, project itself, they do based upon what they can do, but mostly these are like deployed products who do this and then upstream them. So my request here is to everybody who's using Yachto project to upstream as much as they can in their CVEs. So, you know, that's a collective way to fix it. We don't run like, you know, a separate team that we can afford that here is our security response team, right? So I don't think we can do that. But yeah, I think my request is there for everyone to, you know, upstream their fixes as they find. Yeah, I just want to add that, holy, that it's, as you can see, every release is pretty cutting edge. Like right now you look at it and we're like at 5.2, kernel 5.2.9 or something. And then GCC is like at 9.2 and Bing Utils at 232. So every release we try to keep up with like the closest that we can to what's upstream. And that is kind of a reason why sometimes a lot of CVEs come, right? Because we release with the latest possible version. But at the same time, it's hard to maintain that version for a long time. There's a process issue as well. If you have a CVE for Thud, I can't take it a stable branch maintainer unless it goes into Warrior first. And it is done in Master as well. So you can send the patches, but they may not land in a stable branch unless everything, you know, before that gets fixed. And it's like, who does that? Don't look at me. Just wanted to make. So when you fix CVE in the stable branches, it's still alive. Sometimes you upgrade the version of the package. Sometimes you apply the patches. What general logic you're using? Anything else? Anyone? I want to follow up on this. I think what I was expecting is a kind of mailing list where every company has some packages that they care about. And they share the CVEs and take some work, right? So it's not, we can't expect everybody to do all the work, but rather create that communication environment. I can say, I care about binitiles. I'm going to take care of this. And then the next person takes care of the something else. Right? I don't even see that right now. So that's bothering me. And so you mean like mailing this on the Yocto project? I mean, I'll just say most of the work in CVEs has been done by the commercial partners, mentor ourselves onto this. And a lot is going straight to outsource, as Kim was saying. So that communication is happening, but it's kind of happening directly between the companies. So our secretary lead, Mark Hadley, talks directly to other partner companies. So it's not really happening through Yocto project, but it's happening and we're all contributing back to Yocto project. So we're not using Yocto project as a center for CVEs because there's no staffing, but it is definitely happening because it's crucial to our business and everyone else's business. And so it's happening. It's just not happening in a visible way. So, and I think when we talked together earlier, and I think there is value in making it more visible because people ask and they don't realize a lot is going on with their CVE process, with the Yocto project. They're just not visible. And as you say, there's a security mailing list that's just quiet because as I said, most of the communication is outside of that. So I just want to say there's a lot going on. So what would it take to make this discussion more public? So you take that as a feedback? So a long time ago, I met with some Toshiba folks and they were working on a project to build Yocto using Debian sources. And their motivation was for IoT devices that had long lifespans. And further, by using Debian sources, they could take advantage of LTS Debian and their management of CVEs to provide updates. I admit I haven't looked into that project in a while, but is there any one around that knows the status of that, or if that's even viable, what the challenges are with that kind of approach as opposed to using the project? I think that's the meta Debian layer, right? Can we just use that or wants to talk about the layer? Except, Ken. Yeah, so I think that's an interesting approach. I think it's still alive. They still are following up. You can see some talks they did in this conference. I forgot when the last one was in Europe, I think. So it's an interesting approach for the project in general. We target across build systems and what you end up doing is just not sources, but also patches and this and all. So you end up basically creating additional shim that you have to maintain on top. So it's quite a bit of work rather than just simply pointing to Debian sources and building. The reason how they store sources is also not very homogeneous. In some cases, it's some gate. They'll commit into the gate. In some cases, it is like a tar ball and then they will patch it. If you have to do something like that, I would see that it has to be wholesome. And you cannot go case by case basis because Debian has like 75,000 packages or something like that. So I would rather concentrate on creating a process that we have for the project and concentrate on that one and maybe collaborate with other projects like Debian and others. And that gentleman was saying there to have our security mailing list and what needs are there and approach it from that way rather than sneaking in through into the Debian by building the sources. I think it's not a viable approach in the long term because they do change things. At that time, you might have to change your formats and stuff like that. So I would think that the effort-wise, it's probably a more progressive effort if you had like the security working group or some ways to share your CVEs and bug fixes like that. I think that would be probably a lesser effort also because in this case, you're not marrying two very different build systems. Anybody wants to talk about any experience or anything that you guys are doing with the Yocto projects? Any things that pain points? I mean, I would be surprised if there are no pain points. I wasn't going to talk about pain points, but what I was going to ask is besides the stuff that we've discussed, is there a to-do list you guys have for people who are interested in contributing directly? Yes, so there are a few things that the project is doing. So first, there are meetings. Like, there is a monthly technical meeting that is open to everyone. So I mean, the core developers join, and that's where some of the key new features are being discussed or new issues. I mean, so that's every first Tuesday of the month, something like that. So that's one thing. So I mean, if you are an experienced developer, that would be the right place to actually start and just come and talk. Then, I mean, we heavily rely on Bugzilla. So I mean, the Bugzilla is full of things to do. So at the beginning of the cycle, we do like a planning phase. So we look at everything that exists in Bugzilla, and we try to say, okay, this is going this, we can do this, we can do this, we cannot do. So during the planning phase, you can also help and like pick some task and say, I'm committing to actually doing this task for the next release. So that would be one way. Recently, what we started also, we had people who said that we want to contribute, but this is way too difficult. How do we start? So we have these newcomers. I think we send an email once every week or once every two weeks, where whenever we find something on Bugzilla, we find something that is very trivial. We don't do it. We don't fix it. And we take it as something for somebody who is new to actually come and start. So there is a newcomer list somewhere on the wiki, so which is really, I mean, and like what I said at the beginning, this is for somebody who is just sending your first patch. So I mean, this is like trivial patches, sometimes just like fixing the world or something like that. But at least it's learning through how to actually contribute and how to work with our community. So we do this on the two different sides. So for the very first few patches or if you want to contribute and really contribute to the release and be part of the technical team. Does it answer the question? Every Thursday morning, we have a weekly bug triage. So whatever failed at the builds or what bugs came in, we look at it and say, who's going to fix what goes into the newcomer's bucket and things like that. Yeah, that's actually even more than that. We have a triage team, a squad team, which is actually rotating. So we have like a SWAT team, sorry. Okay, yes, but that, so there's basically five, six people which we rotate every week or so. And the role of these guys, just to monitor for build failures and report and try to do like the first personality. So if you have someone and if you want to actually help, the project is very helpful. It's testing master and master next. And we need someone. I mean, there are issues and sometimes it's not an issue sometimes. So we need someone to do actually the first step triage. So it would be one way. So if you have people that actually want to start engaging and work with that, you can also come to Armin and I and we can actually help you with that. You can also look at the Yachta project weekly and then the planning is there and the features for release are there. And actually it gives you a summary of the bugs that are planned for every milestone and what their priority is basically. So if you're interested in fixing something, you can go ahead and just fix it. How many people use like the SDK or the extendable SDK or depth tool? Okay, because we're kind of looking for somebody to take on the technology to maintain it because we don't have maintainers for that. So that would be a good place to get involved if you like. Otherwise we may drop them on maintainers. Yeah, I don't know if I can maintain it but we certainly put depth tools not able to run menu config for the kernel as well. So it's a different flow. Some people like depth tools. Some people like BitBit. It works for some. And I would just say that I wouldn't like to die. No, it's useful, it's definitely useful. That might be a topic that's a bit petty with the rest of the discussion that's happening here but you mentioned if we have pain points. Before talking about pain points I just want to mention that I enjoy Yocto a lot. I'm always amazed that it can build for seven hours and it actually delivers something that you can run on your system. To me it's still incredible. Anyway, so there are many pain points on the flip side. One of them that I stumble across a lot is a state mismatch. And I've found in the manual that there is a big minus S print if they'll try to explain why it's not matching but I'm having so much difficulty finding information in there and I'm not sure how it works, how it traces back. If it goes to the estate mirrors to find possible targets and which estate it finds as the possible the nearest match and how it compares. But I find a lot of times I will dig for because I'm trying to save hours. When it triggers a build of one or two components I don't mind but when it triggers a build of libc then I'm in for the full rebuild and I don't understand why sometimes so it's kind of weird. I'm just wondering if there are tools that I'm missing or analysis that I'm not performing correctly. I mean a hint that I can give is that we have a CI infrastructure so our builds are being performed inside Docker containers and inside GitLab and each of the runners actually will rsync the estate folder back to a central server that we maintain. So each one contribute and so we have a central, I think we use the estate mirror variables to declare and it's a simple HTTP serve of all of our estate folder and it looks like it doesn't go there to find the diff and the possible candidate. So it's always trying to find something on my machine as a possible candidate and it's diffing against that. So first of all the setup you use is very common so many people use that so I mean there is nothing wrong with the setup. Which tool did you say that you are using? And you are on the recent version of Yocto? So I think you have to also look at what you're building right? So if you're rebuilding libc for example then all bets are off. So the reason is because and I think if you were at Yocto Dev Day there was a talk on hash equivalency. So there is a new feature that is being implemented which basically is trying to weed out some trivial changes you do in recipes and that results in rebuilds. But if you are kind of making a real change say into libc we do not yet have a mechanism to detect ABI change. So we might have fixed a bug really not change any API from glibc ideally you don't need to rebuild it if you're not doing static linking. But we cannot detect that so we have to be conservative. Right so there are tools and we talked about that here as well where you can do like ABI compliance check and if nothing changes you don't have to change your hash. But these are technologies that are a little complex right now so what happens is if you are patching it will go and rebuild all the dependencies. Thinking that the patch you change might have a transit dependency that will kind of you know reflect on to other packages. So that's one reason why it will result in rebuilds. That is a problem. Yeah yeah yeah yeah yeah yeah so yeah so there is a tool called BIFsig. Yes so what it will do is if you use BitBake BIFsig it's going to tell you why he is building it compared to what's the difference why it is thinking that it has to redo that task. So that task will give you actually down to a variable. That variable is causing it to think that it should rebuild it. So that way you can debug it right and I think we do use shared state cache quite a bit on auto builders and so I think certainly we do a lot of testing you know on the auto builder as well where we generate a state cache exactly like you and then we consume it from different builders. Yeah so there's a tool I think it's documented yeah it's called BitBake-DIFsig, D-I-F-F-S-I-G. You can target a recipe you can target a task within the recipe right yeah yeah maybe I think that's certainly possible I mean there are possibly there are issues I don't deny those but I think there is a lot of testing we do on a state. Yeah yeah yeah certainly I think you know you could also go to the mailing list and start a discussion provide the feedback hey you know this is what's happening and we can take it there certainly open for all kind of issues I think we take them yeah yeah yeah yeah certainly I think you know for example you know one of the reason is if you don't use uninative right we have had that kind of problems in the past so we have provided a uninative which is basically a uniform environment right so very important use it you use it for consistency to use a state across say your builder is running a different operating system right say it's running Debian and your host is a Bantu right so using that helps quite a bit for it to not confuse the native packages for example if you are using pocky then it is it is already enabled yeah right so so you already have it enabled so so yeah it could be something you know a bug or something you know well yeah that's good I don't have to convince you then so I was being polite but but yeah you know on serious note you can just reach out on on mailing list and then I will certainly help you out I think there is a nice wiki somewhere in the yokto wiki that explains how to debug the hash mismatch but I think the easiest way if you have two hashes that are like usually says you have these two hashes that mismatch in stamps directory you can find the the signature data this it's like compressed files and then use the tool like the bit bake dash dump seek or diff seek and if you have these two files you can see what's the difference and then you can find find variables that are different between your two builds it's usually like date or or some past name but I mean the setup you use is the one we use on the auto builder I mean we have a shared state and we have like 10 different machines that build I mean keep building all the time so I mean there is nothing wrong of this on the setup side at least that that you know one last question maybe yeah this isn't quite a question directly so much well it's a question but it's a follow on to the signature stuff because I've done the same thing many times because there's three or four of us in the company that understand Yachto and there's 800 developers and so when something happens in the you know that that corrupts or not yet that corrupts their environment so they're seeing three hour build time suddenly it's worth several days of my time to go dig it out and figure it out and a huge chunk of the time I see you know there are recipes that take a long time to build and those are the ones that I tend to see the issue on where I can see okay this is building and it shouldn't be but oftentimes quite frequently in fact the issue is not that recipe and its source or something like that but a dependency and if you look at the bit big bit big diff sigs output you know it basically just tells you oh hey the hash for this dependency changed and so you have to go into that hash file and the corresponding one in both cases and do that diff and then another layer up and then another layer up continuously unwrapping the onion until you finally get to the top where something actually changed or there's a variable somewhere that shouldn't actually be included or should be in the what is it hash base whitelist so my question is has anybody worked on creating any semblance of a recursive version of the bit big diff sigs tool or thought about it so the search order is that will search locally if it doesn't find it it will go if you have mirrors it'll go to the mirrors no it will calculate a hash and if you yes so yeah so if you have built something right and that is on your estate mirror that will have a signature and now you're trying to build it it's going to calculate the dependency signature and then it goes on the hunt for signature match right so if you have built this signature before it'll go and fetch it which means right no there is no nearest there is either match or no match yeah there is match or no match right so because see if one thing changed it could change whole things right so so okay so I think it could be that first time you are there and it's not creating a hash locally right because there is no local downloaded equivalent estate um that could be a corner case we might have to look at it that it's not able to then give you uh a hash that it has calculated to compare against something because it needs two versions to compare right yeah right right so there has to be a policy for it for the access so yeah yeah so so I think that is something probably an item that you could bring up on mailing list or maybe as a we might have some clarifications in there and then it could be a feature we could implement if it's missing it could be tied to the build history I mean we could record in the build history what were the last things that we build and the last hashes so then when you build the next one you know at least what is the last state that you were in so maybe that could be that the build history will keep a history of what has been built so we could also record I mean the hashes that were used to make that build so the next time we build I mean we actually have the previous hashes so we don't have any local estate right and you build it and it doesn't find a match and he hasn't made a change and it doesn't match when it should match and now he can't debug it either right yeah but yeah so there has to be a search policy where you can define preferences and stuff so that's actually a whole different game yeah so it's something that I think yeah so so let me yeah I think I think it definitely could be something we could enhance right it could be something that you are doing that's causing this behavior yeah I think I mean enough people I mean I guess I mean these also were interested I've now I mean now that you've told the story yeah I'm sure you really made we believe yeah that's an interesting situation so so so when I when I when I debug these and I debug them a lot generally what happens is I'll have a build I'll basically start with no at local estate cash and remove the temp and do a build and then basically when it gets to the very absolute first thing that it tries to build I basically take that and generate the signatures and try looking through the signatures because if you if you I think if you do say just one it'll tell you all the things that's using to calculate the signature and if you look through those oftentimes you can see oh I've got a path in here that's a local path or something of that nature and that's what's causing it and so I need to go and whitelist that that's usually my first step is basically clear out the workstays do a build and see the very first thing kill the build and generate the signatures for that and just just look at the look through the signatures that it's using to calculate the hash and usually often you know you'll find oh there's a straight time date or yeah basically basically what happens is is it's a dependency tree right and the the one hash at the very beginning cascades through the whole system so starting at the very beginning the very first thing that starts building that it fails yeah yeah basically start at the very beginning and just just generate the signatures look through the signatures and you'll often find there's a date tag or something something of that nature that that is triggering the whole thing yeah you can also tell bit big to just do the set scene tasks it makes you look really smart in front of your coworkers well you want it to run the set scene tasks so it does all the estate it can and then dry run after that and you'll see everything that it would have built that wasn't an estate that way you don't actually do the build on their laptop but you run as much set scene and that way you can figure out where you can at least pairs it down a lot right I figure what the option is but there is a manual okay so we any any last questions or are we done one more okay okay is somebody using reproducible builds and what's your experience how much reproducible packages do you are you getting I'm not using them but I'm writing test cases for them I think the plan they're very close to passing so I think the plan there is to write test cases that will run on the auto builder to make sure that core image minimal and core image seto are reproducible and Richard really wants to get that in for 3.0 right now it's me and Ross has been helping to well it's really important for hash equivalence and I'm not sure what the other use case oh I know people have other use cases just because they want binary reproducible builds you will have issues right now with the infrared in phrase there but we have to patch packages right now in our recipes you know there are packages which use date and then time and stuff like that so I can't tell you like you know a number what percent it is but right now I think you can't build an image where you don't have one package that is not I mean that is you'll have at least one that will fail so yeah so but I think it's it's progressing I think once the testing goes in you know what what Josh is doing I think that's that's actually pretty good there after I think you know we can basically enable it on world builds and you know we can get a lot more information on what all is failing basically right so yeah so we are working on it um we want okay good so it's always like that we don't have any questions and then we can't stop anymore so thanks everyone again I mean we really like I mean unfortunately I mean it's not a lot of time but we are I mean most of these guys I mean this conference all the time so you can find us at the york to booth somewhere anywhere so you can always come and ask questions and you can obviously always use a mailing list so thank you for coming today and yeah thanks for your feedback about the project thanks