 Could you raise your hand if you consider yourself to be relatively new to Yachto? So you've only done, say, one or two builds or maybe one or two products, okay? And then raise your hand if you consider yourself to be sort of an old hand. You've done this quite a few times and you feel pretty comfortable and you're just, okay, well, this is good. There's a good mix. Okay, so for the people that raised their hand and said they're relatively new, I'm gonna move a little quick, but at the end is contact information. So please download my slides and don't hesitate to email me or tweet at me or whatever your preference is. I'm happy to answer questions. So thank you for coming and a little bit about myself. So I work for Intel and I work contributing full time to the Yachto project. My chief concern at work is identifying pain points for customers and then either resolving those pain points or figuring out what needs to change upstream to fix those. And then I work with both internal and external customers. And the goal is to make everyone's development more productive and faster. So I'd like to share that goal with you here today. Before I get started though, I'm gonna plug the other talks that are going on for Yachto, there's still a few left. Tomorrow in particular, Tim Orling's talk, I highly recommend. Cover some of the stuff I'll be talking about here, but in much more detail. And then also, Yachto Project Developer Day, which is gonna be on Thursday. I'm actually gonna be teaching a session there on package feeds. And that's gonna go into depth in a lot of the things that I talk about here today. So let's get started by taking some time off and actually just talking about the system and the benefits it has. So one of the things I realize interacting with customers is people don't really understand why they're using Yachto. And what I hear said over and over again and what I really like is that Yachto is essentially a line drawn in the sand of open and bedded. So we're picking certain revisions, certain versions of software. We're drawing a line across those. But then we're testing those, and that's the part that people miss. So we have, underneath the link to Paki is a link to our continuous integration, the auto builder. Now you don't need to know anything about the auto builder. But what you need to know is that we are, on a daily basis, testing four different architectures on something like six different distributions to do the build. And these tests are being sure that the versions of software we've picked for our release are going to be tested for compatibility and performance. So by choosing to use Yachto in one of our releases, instead of just rolling your own open and bedded distribution, you're going to get the benefit of all those tests that we're doing. And if you glance at one of the links there, there's downloads.yachtoproject.org. We happen to have a Beagle bone and want to test that hardware. We provide you with SDKs and images. So you don't have to build it. You can just go pull it down. Obviously, we have a bug tracker to keep track of all the bugs we find when those versions of software do not build correctly. So you can go take a look at our never-ending stream of bugs. And while I had a whole bunch of links up on the screen, I wanted to add that link to layers.openembedded.org. Not enough people know about this. So let's say you have your build and you're looking to add a piece of software. You don't want to have to write that recipe yourself. So someone must have written that recipe already. Your first port of call is layers.openembedded.org. Type click on recipes and type the name of the software you're looking for. That's the first place you go, the next place you go is Google. And then after that, I would say you should go to DevTool, but you'll be writing your own recipe. So I was taking advantage of having links on the screen. So that's a general overview. And this is a general overview of some of the concepts we talk about. I think for the most part, even the people who are relatively new to Yachto in the room have an idea of what these concepts are for. The concept of a distro, which abstracts out things like the UI and which Jalib C you're using. The concept of a machine that abstracts out the hardware details. But the question I'd like to answer today is how we take these concepts and build useful software with them. So we take these concepts and we build off of them different layers. And I'm using layers in the plural there. It's very important you remember that. Because the layers that we build aren't inherently structured with distros and hardware, but the idea is that you can use this abstraction layer to form a set of layers that will make your life easier down the road. So I think we all have a basic idea of these concepts. But one of the things that people tend to do is they tend to lump everything into one giant layer. And I see this all the time. And it's a relatively quick way to get your build started. You put all of your distro information, I'm using an RFID example here. But you put your distro information, all the hardware requirements that you have, and all of your software all into one giant layer called my company's layer, and then you ship your product. And while that is a quick way to do it and will certainly be effective, I'm here to stop you from doing that. Because what you're doing is you're making future use life painful. So you don't see it yet, but down the road you're gonna be kicking yourself for doing this. What it's gonna make, it's gonna make upgrading your software really hard. It's gonna make upgrading your hardware or changing the product that you ship much more difficult. And because you're all going to upgrade to the latest version of Yachto at some point, it's gonna make that really hard. Cuz no one here is on Dizzy, right? Everyone's running on Pyro, yeah. Okay, so you don't have to raise your hands. Okay, so the goal is multiple layers. That's one of the ideas I wanna get across. Make sure you're thinking about future you. Make sure you're thinking about what it's like when you do finally get to upgrade to 2.4, to Rocco. Think about what happens when you need to rethink your hardware. Because that's gonna happen. Your boss is gonna walk in and put something new down on your desk and expect it all to work in the layer system you've built. So if you have everything chunked up into one layer, you've just made your life really difficult. So my example here, as I said, is just an RFID product. It's an RF product that you're making. But when your boss comes in and drops a web kiosk on your desk and says, of course, this is all gonna work in your layer scheme. You can say yes if you separated things out. So when you separate out your distro layer, it makes it easier for you to keep a couple of different distros in there. Maybe you have a frame buffer and an X11 layer. So a really complicated UI and a really simple UI. If your hardware uses different architectures, or even if you just don't wanna have those hardware is mingling together. So video really doesn't have anything to do with RF. So separating those out into different layers makes it easier for you then to distribute these. So now I know you might not be distributing your layers to customers. But think about the internal distributions that you do. Working on your team and handing this off to your teammates. You want these layers to make sense and you want them to be easy to modulally remove and add. Software is another great example. Obviously if you're gonna be having things like Python, living with things like C code. If they don't have anything to do with each other, those separations of concern mean that you can ship the manufacturing people just the thing they need. Or ship the QA team exactly the layers they care about. The idea here is to make layers less painful to change by not having them all in one group. So basically you're taking the requirements and allowing the requirements to change while keeping your layers in a manageable state. So I've talked about one of the biggest pain points I see. Which is everyone throwing everything to the same layer. But over the past few years I've collected a set of tools that I've learned about. And a set of really tips and tricks that I realize will make everyone's life easier if we talk about them more. And a lot of them are buried in the complexity that is the Octo project. So the first one I'm gonna talk about is, may seem a little simple. But I think it's a concept that everyone needs to rethink. So, Yachto generates some image. Something you're going to put onto some media and try to boot off of. Traditionally we've made an ISO 966O basically a bootable CD. And I'm sure most of you out there pull some vendor layer in. And that vendor layer is then gonna give you some customized piece of bootable image. Whether that be an SD card or NAND. You're relying on that to do just the right thing for you. And in the past it's been really hard to then change that. So if you wanted to add multiple partitions or heaven for fend if you were ever try and do some sort of layered architecture where you're layering read only and read right on top of each other. That used to be really difficult. So there's been a tool that we've been working on for a while called WIC. And WIC stands for Open Embedded Image Creator. So I admit those letters are a little confusing. But it is a catchy name. So what WIC is gonna do is WIC is gonna read from what we call a kickstart file, a WKS file. And this is gonna allow you to generate custom partitions and custom media that you're then gonna burn to. So for example, you might have your manufacturing expects to just get an SD card from you. And some of your less careful customers may boot off of SD cards in the field. But you also might wanna boot off of NAND or boot off of NOR and have your file system in NAND. These kickstart files allow you to separate these ideas out. Very cleanly, and then reuse them. And this way you're not writing a custom VB class every time you want a new type of media to boot to. So I really encourage you to go look at WIC. There's lots of really interesting little tricks that it can do. For example, if you've generated a WIC image, there actually is a tool inside of WIC that will let you copy files into that image or remove files from it. You can also just LS, a directory inside your image. So it allows you a level of introspection into that thing you just generated. Whereas before it used to just be a black box. That you deeded to some location and hoped that it worked. And when it didn't, you thought, well, hopefully something is cropped. Let's try again. And speaking of DD, WIC takes advantage of BMAP tools. Can I get a quick raise of hands of everyone who knows what BMAP tools is? That is so sad. Okay, so I'm really glad I'm talking about this. So DD, as you all know, is not a very intelligent piece of software. And I'm not coming down on DD. DD's wonderful. But its job is to take bits from point A, copy bits to point B. Did I just summarize DD pretty well? Okay, so BMAP tool realizes, well, you're gonna copy some useful bits from point A to point B. But at some point, you're gonna copy a bunch of zeros. Why don't I just skip that? So it does a sparse copy, basically. So WIC takes advantage of this. By default, I believe by default in Rocco, we are now spitting out .wik files and .wik.bmap files. So my apologies for adding one more extension to our lives. But by using BMAP tools and pointing it at that WIC file, it'll look for that same file name, .bmap. And it will, basically you're talking an order of magnitude faster. So go home after this, take a 16 gigabyte Ubuntu image, and use BMAP tools to put it on an SD card. And then you will never, ever, ever use DD again unless you have to. Okay, that's all I'll say about that. So another thing that I find customers don't know about or they know about and they don't use because they think it might be too hard or they're not sure exactly how it works, is SDACache. One of the biggest complaints Yachto has is that it takes a long time to build and it does. If you put it side by side with something like build root, you are gonna win with build root for speed every time. But what a lot of people don't understand is that Yachto is doing a lot of things behind the scenes to make a robust build system and that's what's taking all that time. So because we think those things are important, like specific debug flags and build history and package management. We're doing all of that, but we realize you need to speed up your build. So we've introduced SDate. SDate by default is gonna show up in your build directory. So my first piece of advice is remove that from your build directory. Put that somewhere else. We have variables that let you put it wherever you want. Me personally, I put it in opt Yachto and the idea here is being you might have ten build directories someday. Especially if you have a very aggressive sales team. You're gonna probably have ten different build directories at any one time. Make sure all those build directories are taking advantage of downloads and taking advantage of SDate cache. You don't ever wanna download something twice. And for that matter, if your whole team is not sharing a downloads directory, please, please, please go do that. R-sync, NFS mount something, whatever you have to do, it saves you that amount of time. And likewise, you should be using SDate mirrors. So, and I'll talk about these in another slide. But one of the things I do to make this much, much easier for myself is site.conf. So site.conf is just a configuration file that Bitbake will look for when it starts up inside the conf directory. So what I do is I have a script that starts up my build directory every time for me and copies of my site.conf in there. My site.conf sets my build, my download directory, and sets my estate directory. So I know I'm at least getting that bonus every single time. It also sets my mirrors. And that's the next thing I'm gonna talk about. Everyone thinks about estate mirrors as our team probably won't benefit from it. Or I don't know how useful it is. It is extremely useful. Last week, I was talking to a documentation person on our team. And he was having problems with one of the builds, but he was building 2.4.1, one of our releases. I showed him the link, I showed him how to set it up, and his build took about seven minutes. Because he was just pulling stuff that we had already built on the auto builder. There was no reason to rebuild any of it. It looked at the shaws, they were all correct. And it generated a root file system for him. So at the top of this slide, I'm calling out something really important. And I think this is something that a lot of people don't know too. So when you're basing your project off of Yachto. Yachto is working on a branch, right? We have our master branch, and then we have, for example, here we have our Rocco branch. That is a constantly moving target. We do have an estate that's populated at estate.yachtoproject.org for that dev branch. So if you're working on there, give our auto builder some time and it'll update that. But if you're doing a release, you wanna be using the tag that we've tagged for our releases. So every release, even our point releases, have tags associated with them. When I said back at the beginning of this talk that we choose a bunch of versions and we guarantee they're gonna work and be compatible and all that, only if they're on that version right there. So if you're pulling off our dev branch, well, we're testing it out our auto builder every day. But really, you wanna be pulling off of that. 2.4.1 is a great example. And you should be doing the same thing we're doing. When our auto builder finishes a nightly build of our dev branch, it throws all that estate information up on a web server where anybody can use it. When you try and go do a build off of, for example, 2.4.1, estates gonna go look at that mirror, see that all those shots are the same. And just like my coworker, it's gonna build in no time at all. Now, obviously, that's not gonna be the same every time. You're always gonna be changing things. But this will save you an exponential amount of time if you just need a base image and then need to go from there. Also, I was talking to some of the group and they wanted me to mention that if you happen to be working for a company that happens to be one of our partners, we are talking about possibly making that available as a benefit to some of our Platinum members. But it also brings up the fact that you should be making that available to your customers. If your customers are doing builds using your layers, they should be using estate. If your QA team is doing it, your QA team should be pulling from estate cuz they're not changing anything. And your manufacturing team if for whatever reason they're doing a build from scratch, they should always be doing a build off of estate. And again, please feel free to contact me via email if you have questions. If you run into any weird problems with estate, there are the people that wrote estate are still on our team and we still have the tools that we use to build estate available to us. So we can answer any questions that you have. This is my required kit and slide. So along the same lines as estate, there's another way to speed up your build. So I was talking earlier and saying, we're all used to the churn and burn of we burn a card, right? So we're using DD, which hopefully all of you will go home and never do again. But you're using DD, you're burning something to a card, you're putting it in a board, you're booting, it doesn't work. So you take the card out, you change your software, you put the card back in, burn it again, put it back in the board, boot it, it's missing a library. Take the card out, if you're smart maybe you're SCPing the library over. But what you're missing is that Yachto is generating packages for you. It's generating, in this case, the example I'm using is RPMs. It's generating all the RPMs that you need for your software. So if you're doing a build and your build needs lib something, you probably have a package called lib something sitting in that RPM folder. So Yachto has the ability to run something called package index, I believe. Bitbake, yes, bitbake package index. So by indexing that folder, by creating what RPM is gonna look for when it needs to parse through all that information, you essentially have a repo. And if you look at the steps that I've added, there's two steps to add package management to your board. Two variables that you need to set. The other variable which is optional but I think is really important is the PR server host. This, as you change your software, it's going to auto increment the versions. So if you are pulling these RPMs, those RPM versions will increment. So the RPM package manager will see the change. And then the bitbake PR server. So you actually do need to start that server up. Once that's running, you're gonna be generating RPMs and every time you change your software, those RPMs will increment one version. So if you've then shared that folder, and in this example, I use an example of both Lighty and also Twisted. I use Twisted because it's really quick, it's simple, and I can stop it or start it sort of willy-nilly. But you could have a permanent setup with something like Lighty or Apache. And share that RPM folder. When I say the RPM folder, all I'm talking about is in your temp directory, there's temp deploy RPM. That's the folder that you need to share. And if you look up at the top, there's the package feed URIs. If you set up that package feed URI, when you generate your image, it'll be automatically populated in there. So you really don't even need to change anything on the board. So by creating that package feed, sharing that folder on some sort of web server and running BitBake package index, you've now saved yourself the trouble of pulling that SD card and copying something onto it. Now on the board, you can just say RPM install. So a great example is when I work with customers, I get a lot of customers who are brand new to Yachto. And they have a product that they need to ship last week, generally. So what I do is I create an image for them, and that image is never gonna be enough. So what I do is I create the image that they think they want. And then I run BitBake World. BitBake World, now to be fair, you do remember who I work for. So I'm doing this on a rather beefy server. But whatever's in their layer, any of the software in their BitBake World will then compile, it will create a package of that. I take that and I put that up for them to be able to download. And then when they email me and say, but I'm missing Valgrind. I just said, RPM install, Valgrind, you're done. So this saves me an immense amount of time. And it can save you an immense amount of time, not only in your personal development when you're doing development on the board, but anyone at your company or anyone externally who needs to very quickly install software. Sorry, I missed one thing. All the way down there at the bottom, RPM and O package. We are currently testing this with RPM and O package. So yes, you can use D package. And yes, we will accept any of the bugs you find. And no, I will not be surprised when you find a bug there. Okay, so the next is a question that I get all the time. I get this pretty constantly from customers and that's, I just built my file system, I booted my board and there's a file on here. Why is that file on my board? I didn't put it there, something put it there, but I don't know how to find it. So the first utility that I'm using, their OE package data util, a lot of people don't know this. This is my first stop that I go to when I need to answer that question for them. And what that's gonna do is hopefully it's going to output the name of the recipe that caused that file to populate on the board. So in this case, I'm pointing it at bin SH. And so hopefully in a simple core image minimal build, this will say busy box. But you can point that at any file and it will do its best to introspect your build and figure it out. This is essentially the thing I do before I get grep. Because get grep is the natural thing to do when you're looking for how did this get in there? Well, the file must be sitting somewhere in my layers, right? So those are the two most common things to do. Again, if you're using package management, you have DNF what provides as well. So we switched recently from smart to DNF and the difference is huge. DNF gives you a lot of introspection into what's on your board and why. But keep in mind that when we do a build and we create a file system, a lot of this stuff is done dynamically. There are just a couple of examples in NIT scripts, or a great example of a way where somewhere in RCNIT.D, it's generating a file and plopping on the file system after you booted. The system can't really tell you that. If you dig into the guts of Yachto, you'll find that there are actually root FS post-process commands that can be run. So if someone has tried to sneak something on your board by using a root FS post-process command, these won't solve that problem. And so for that, I have instance number five, which is IRC. One of the things I love about working on the Yachto project and I love about the group I work with is we're all pretty friendly guys. The IRC channel is a great place to ask questions. We have one of the more friendly open source communities, in my opinion. So please, if you find something and you don't understand how it got there, you've used all these techniques, you really just want to know, post it to the chat and we'll respond. Another common request is not just how did this get on my board, but rather, I know how this got on my board, I want to change it. So think of any config that you've ever added up on your board and realize, okay, now I need to go write a recipe to change that config. So my Apache config is wrong, now I have to go write a recipe, I have to go figure out what that means and we have a tool to do that for you called recipe tool and the command is append file. So Apache config is a great example. If you call recipe tool append file and point it at that config file in the tree, it will generate the recipe for you. It's a relatively simple recipe, but you'll be able to throw it into your layer and you've edited that config file in three minutes instead of 15 or 30, depending on what kind of day you're having. The last one which kind of flows naturally from appending configs is the dependency tree. So when you're trying to figure out why stuff is on your board, you'll often be interested in how the dependency tree has worked out. You might know that Busybox put that on your system, but what would Busybox on the system? BitBake has an option dash G that will generate using GraphViz, a dependency tree for you. And coming in 2.5, but not out yet, is oedepends. Which will then save you from having to look at that gigantic dependency tree, and instead will let you introspect specific parts of it for you. And I think this gets at one of the things that most people miss with BitBake, and that's that BitBake is just a normal program like every other program you use. So one of the things it has is a dash H option. And I know it sounds silly, but a lot of people don't think about it. So there are tons of options inside BitBake that I use every day that people have never heard of. So I put my favorite one up at the top because it reminds me of lunch. Whenever I go out to lunch and I'm gonna do a bill that I know it's gonna take forever, I use dash K. What dash K is saying is, I'm gonna start this bill, and eventually it's gonna air, but I'm gonna keep building everything I can for you so that while you're off at lunch, a lot of work gets done. So that I use pretty much every day. We already talked a little bit about the graph vis option. Hopefully a lot of you know about the dash E option. The dash E option will output the BitBake environment. So if you have questions why certain variables are set the way they are, that's a great tool. One of the mistakes that I see people using every day though is they're calling BitBake dash E and they're piping that to grep. Because they wanna grep through the results. But you're losing a whole bunch of history around that variable. So instead pipe it to less and do your search with less. And what this will give you is not just that variable that was set, but the variables around that and give you some context into that. It's a common mistake. Another option this is dash capital C, which I've made super bold. So everyone can remember. Sean actually mentioned this in his talk yesterday. So dash C invalidates a stamp. So we talked about estate cash a little bit. And I was mentioning earlier, estate cash can have its own talk. It's a really complex system. But the idea is that it's generating a hash of the input and tagging that to the output. So for example, in do unpack, it's going to look at the input to do impact, do some sort of magic hash on that, and then look at that hash. And if at any point that hash has changed, it's gonna redo that task. So in this case, do unpack. So if I know I want to undo do unpack, that's something I need to happen. If you just use dash capital C, it'll go back to that task, undo the hash. And then undo everything afterwards, right? Because all those other hashes are now invalidated. They're all based on that do unpack hash. But you could do that for anything. You do it for do configure, do install, pick your step. So it's much, much more fine grained than things like clean or clean estate. It's a very precise tool. Speaking of tools, there are lots and lots and lots of scripts in the Scripts Directory. And one of my tasks for this talk was to go through them all and find the ones that I liked the most so that I could talk about them. So Bitbake Layers is a tool that's actually been around a while under different names. And it's a great tool when you're building layers or searching for layers. It has a couple of options. I'm not gonna get into details on it. Go take a look at it, especially when you find yourself building a layer regularly. You should be looking at DevTool and you should be looking at Bitbake Layers. DevTool is very much a talk in its own. DevTool is a class in its own. But there is a good talk by Tim Orling from last year about that. He goes into details on how to use DevTool. Sean talked about it a little bit yesterday, but if you don't consider yourself a full time kernel developer but you need to do some edits to the kernel, DevTool can really be a lifesaver. So if you're just looking to edit the DevConfig and you just know that this driver right here is broken. And you need to tweak that C code until it's not broken. DevTool can give you an environment inside the SDK to run make and to run and to build your kernel. And it'll allow you to do that in such a way that you're not having to develop your own cross environment. You can use the SDK's cross environment. It is also a really handy tool for generating recipes. So once you've finished with something like making a small edit to the kernel, rather than having to make your own patch, DevTool can make the patch for you. And again, I'll point you to Tim's talk. It's a great overview. And there are tons of documentations out there on the web as well. I talked a little bit about OE Depends. That's coming on later down the road. I think that'll be in 2.5. Bitbake dumpsigs and Bitbake diffsigs. This is another one of those really underused scripts that I think is super important. So I'm pretty sure if I asked, everyone would raise their hand. You just changed something in your recipe. And Bitbake went ahead and bitbaked 25 things. And you're thinking, whoa, I made this one change. What are you doing? So what's happening is that change has caused different estate hashes to invalidate. And to know how those are invalidated or why those invalidated, this is why diffsigs and dumpsigs exist. They were actually used as the tools to create estate, to test that it was functioning properly. But you can use them to then look at those sigs and see what changed in there. So dumpsigs will take a sig file, which Yachto has generated. And it'll dump all the information in a very readable format so you can see, these are all the things. It's basing its hash on. And then you can use the diff tool to look at the two, beside each other and see what's changing. So you can work out whether or not there was a dependency change somewhere hidden in there, or an R depends change, hidden in there. So these two tools are extremely useful. Let's see, and build history diff. Build history diff is another good one that I think people don't think about a lot. Changes from build to build are hard to track. And that's why build history exists. It exists to try and help the quality of your builds be better. So build history diff is simply a way to take two different build histories and see what's changed between them. It can help you not just in QA, but also in just trying to introspect your system and see why things are being built the way they are. The last two tools I'm gonna talk about are required in any open source project. And those are the create pull request and send pull request. The Yachto project is heavily dependent upon community involvement and like I said, I think we have one of the more friendly and inviting communities out there. So I really encourage all of you to go out there. If you're brave enough to go look at Bugzilla, please do. If that's a little much, just find a part of the system that you're interested in working on and shoot one of us an email. Because we're always willing to take contributions and we're always willing to help. We're always willing to guide you through the process. These two scripts are just a couple of simple examples. We have people that are coming onto our team internally all the time and these are scripts that we're constantly improving on to make it easier for you to put patches into the system. And we also realize that this is a really complex system. I think I actually counted yesterday's and yesterday's patches considering that this is going on, we're still over 100. So there was over 100 patches to OECore yesterday, even though there's an embedded conference going on. So on a normal day, there's somewhere around 500 patches getting sent into the mailing list. So we realize this is a complicated process. It's hard to wade into this. But we as a community are willing to help. The OE community, the OCTO community, we're willing to help and guide you through that process. So with that, I'm going to remember that it's getting late in the day and you're probably trying to decide which brewery to go to. So the last two things I'll mention before I take questions is the bug triage and the technical meeting. These are two meetings that are held. The bug triage is actually every week, Thursdays at 7.30 a.m. PST. And the technical meetings we hold on the first Tuesday of every month, 8 a.m. Sorry, that's 8 a.m. PST. So the bug triage, you're always welcome to join and just listen in. If you're just curious how the system works and how bugs get pinned to different people, like who gets what bugs and what we work on. And you'll actually get some introspection too into where OCTO is going and what sort of features we're working on. And the technical meetings are a great place to come ask questions. So all of the core developers will show up every month at this technical meeting. So if you don't feel comfortable chatting on IRC or it's hard for you to put the idea or the problem you're having into an email, come to the technical meeting. We are all available to answer any questions. And I am amazed every month at how few people take advantage of this. It is just free advice from the people that basically wrote all these tools. And we're happy to answer your questions. So also at the top is my contact information. So of course you're always welcome to email me or yell at me in public if you find something you don't like. So that's all. With that I'll take any questions that you all have. GCC cache. Is that class in Yacht? Okay, no I actually haven't heard of GCC class. Is there a part of Yachto where you've seen like a BB class that references it? Or ping me offline. Well yeah, we'll figure that one out because that is actually using one. Sure, so no it's a good question. So the question is basically he's inheriting from CMake and there's a do install being called and it looks like it's being called more than once. And so the question is around, how does do install get happen? So one of the hard things about Yachto and it takes time to learn how to dig through this is the fact that really everything is layered through with this inheritance. So CMake probably, I don't have the code in front of me right now, but I'm pretty sure CMake inherits from some other class. So you have to walk that class hierarchy to see how all those do installs actually happen. So I'll give you just a basic example. You can have a do install in a BB class that you're inheriting from. But that do install may just be appending a do install because the do install might actually come from a class much further up the chain. Now so if you don't have an underscore append then you are overriding the do install. So if you have a do install you can yourself say ignore whatever do install does, I want you to do just this. Exactly, if you were to write that and then just, if you were to write your own do install and your recipe and just say echo hello world, do install is gonna echo hello world and be done. So and just so you know the other ways that you can introspect that too is to do these builds and then use build history diff to see what change. So as you change your do install, look at the differences between those two builds. The diff tool is just a side by side comparison so you'll see the exact changes that happen. Sure. So the question is about running Yachto or building Yachto using Jenkins. That is a great question. And there is an entire talk tomorrow that you can all go. So Tim Orling has a talk tomorrow. Called living on master. And in his talk he's gonna detail a step by step evaluation of using Jenkins to do builds using Yachto and he does it end to end with hardware. So it's a great, I don't know if there's any documentation today about Jenkins, I don't believe so. But one of the things that we're doing too is we're rethinking the auto builder. We're sort of retooling it. It has been a long time since it's been retooled. So we're retooling the auto builder now and we're thinking about ways that we can use Jenkins because we know a lot of people are building with that so we'd love to have a quick start for Jenkins and Yachto. As with most things, we're usually constrained by resources. But if you are interested in Jenkins and just CI in general, I highly recommend you go to Tim's talk. Tim has been doing real-time Linux builds on master doing builds on, I think something ridiculous, like 10 different builds every day. So that talk will be an amazing and a lightning experience for you if you're building with Jenkins. Thank you. So there are others. This is a great question. So this question is essentially about versioning your layers which is a good way to put it. When we talk about it in our meetings we tend to refer to it as a setup tool. Because at the end of the day you wanna set up your build directory with a set of different layers and those layers have to be out of a certain version. So you wanna check this tag out of this layer and this tag out of this layer. It's a very common problem. Yeah, so repo is one of the main ways. I know FreeScale, the community BSP uses repo. There is inside Yachto a script called combo layer which is what some of our team members use internally. And I know a lot of people in fact, actually could you raise your hand if you have made your own tool to solve this problem? To basically check out a whole bunch of layers. Okay, yeah. So this is very common and we realize this is an issue. One of the problems with this issue is that everybody does it a little different. So what works for one company, I mean the simple case, a script that does the simple checkout is an interesting two hour endeavor and you're done. But the complexities that lots of companies face is that they need a tool that not just does that simple thing but does lots of other stuff as well. And so that's why you end up with everybody rolling their own. So we have been talking about this and trying to figure out ways that we can develop something that works for everyone. I should also mention that Wind River has a tool that they use to do their setups. I think if you just Google Wind River Yachto setup you'll probably find, I don't see Mark here. Or just find Mark Hatley. Yeah, pardon? Yes, oh okay. Yeah and currently, so the tool that Wind River developed is currently lined up to hopefully become a standard for open embedded but that is one of the things that we struggle with. So thank you for asking. Anybody else? You can, this is really, this is dependent on your setup. So the question is basically he's trying to populate an estate cash which I applaud you for sir. So he's trying to generate an estate cash and he wants to know on his build server. He's using Jenkins. Do I wanna do this every time I build or do I just wanna have a special case that only generates estate cash and that's its whole job? You can do either. It depends on your use case. So we do it with our builds because what we care about is whether or not the build succeeded and that's our estate cash we're gonna go off and we actually only generate estate caches based on success. So if we have a failure we don't store that estate cash. For our releases, obviously those releases once they succeed we pin that to two dot whatever and that gets, so it's really up to your use case. If you did have a use case where you needed estate cash more regularly you could do that. You could, I would however recommend if you're going to run a build that just populates the estate cash be sure that that's been verified first and as long as that's been verified sure there's no reason not to. Sure. Yeah so the question is about estate cash with multiple builds and whether or not there's conflicts there. So I did not write this system but as far as I know from my experience we're currently doing builds on our auto builder. Each auto builder has three builds happening at the same time and they're all pushing to the same estate cash. So I guess I'm answering your question by saying here's how we test estate cash. We have six servers. All six servers are building three builds at the same time and all three builds are populating at the same estate cash. So if there's a bug we will find it. Anyone else? So the question is about estate cash and really the fundamental concept of what Yachto is doing. So you're building from scratch with Yachto. What's the point of estate cash? So I think to answer that question I would say estate cash has its uses. We do builds from scratch all the time. We don't rely on estate a hundred percent but that being said there's lots of instances where building from scratch is an ideal. So I'll give you an example. I have teams that I work with that are building on laptops. So yes, for their production they do wanna build from scratch. They do wanna ensure that that's happening but it's just unrealistic to expect their two cores to crank out an image in any respectable timeframe. So really estate is the only solution for them. And at the end of the day it is really it's meant to be used circumstantially. You should figure out whether or not it's right for you and use it. But there are other ways to speed up the build. If you go and look at your bitbake.com for example you'll see a whole bunch of debug flags in there. And so if you really just need to speed up your build if that's all you're looking for there are lots of other ways to do that besides estate cash. But hopefully that helps some. Yeah, Dennis brings up a great point. There's often a separation of concerns between people who are building as build engineers who needed to run on a certain platform and ensure it's configured properly and then there are application developers that just need to build so that they can write their Java app that sits on top of it. And those guys really don't care whether or not the G-Lib-C is building and getting built correctly. Sure, excellent. Thank you, Dennis. I'm saying no, don't do that. I heard from the guy. Sure, no. So the question is essentially that there are plenty of places on the wiki you can go and find best known methods of doing things with Yachto. And there are very few places that you could go and find the worst known method of doing things with Yachto. And I think... That's what they always choose. Right, and that is what people choose. And to be fair, that's why I'm here giving this talk because I have seen people cram everything in one layer and I've seen them make all these mistakes. So no, it's a good point, actually. So we should have a part of the wiki that is devoted to please don't do this. And to be fair, that is partly what I hope this talk will become. I hope that I can take these slides and then turn them into a wiki where people can go to it and say, okay, these are the things I shouldn't be doing. Here's why I should be doing the things that are in the BKMs and the best known methods. I will ping Randall Monroe and let him. Any other questions? I think we still have five minutes. If anyone has any questions. No, please do. It is, yeah. That's the email address right there. Yeah, no. Sean's comment is great. Yeah, absolutely. As you guys run into things that you realize aren't best practices or you're doing something and banging your head against that, please email me. Let me know. I will put it up on the wiki for you. Or I'm happy to do documentation. Just ask any of our documentation people. As soon as I have time. Any other questions? That's a great question. So Margot's question is, there's a difference between estate mirrors and estate directory and it's a subtle difference but it's really important. The estate directory is for you and your build server, your workstation. The intention there is if you have several different builds on your machine, that is the first place it's gonna look for estate cash. But for sharing with other developers across multiple machines, that's where estate mirrors come in handy. So what you should be doing is populating your mirror from your developers or really what you should be doing is populating it from your build servers, estate cash. So your build servers, estate there should eventually turn into the estate mirror. The idea being that that mirror should be of at least a relatively known good build and that it should use some protocol other than a file system protocol to go get it because you know you're gonna have multiple developers hitting that machine and requesting estate hashes. So if you go, it's on the slide way back there but you guys can go back and look. If you go back and look at estate.yachterproduct.org you can actually browse that and you'll see that it's just a giant hash and so it starts with zero zero and you can drill your way down through that hash and so that means if 10 developers are all hitting that at the same time it's being handled by the HGT protocol. So you don't have to worry about collisions or people stepping on each other's toes. So estate mirror is where all your developers point and each of those developers should have their own estate directory and hopefully most of the stuff they need is in there and it'll be really quick and then the stuff they don't have they can go up to the web for. Now please keep going. Now, so definitely, yeah so I would definitely have one person acting as the master and it should be your build server. So your build server should be populating your estate. Your developers can populate their own estate caches. That's a good question. Yeah, hopefully your build server is producing the build that everyone is using and you're just changing the estate of the thing you're working on. So really your changes to estate won't be as valuable as the build servers changes to estate. Excellent, thank you, John. Okay, I think we have one minute. Last chance, oh, there we go. Yes, absolutely. Yes, the reason that it's better to have that in your recipe. So the question is about, you're using CMake files as an example but basically what you're saying is you can put files on the file system any way you want and that's absolutely true. And you really just need to follow best practices when it comes to how you want the file to get on the file system. Just keep in mind that if you want Yachto to be able to introspect that for you and tell you how it got there you should probably have that. If anyone's ever gonna edit it or care where it came from putting it in the recipe may help. But again, just use best practices when it comes to stuff like that. It depends when he does it. Yeah, so for the CMake, I guess you're right, you might get QA errors if you're doing it in CMake. Not if you use it as an example. Yeah, there's lots of ways to turn off QA errors. But no, that's a good point. If you're putting files on the file system with you may get QA errors and you may ignore those QA errors, right? No, that's a good point. Yeah, insane skip for removing QA errors is probably a bad idea. But yeah, hope, right? Hopefully that helps answer your question. Okay, I think we've reached the end. So thank you all for coming. Have a great day. Thank you.