 Okay, let's get started. My name is Thakir. I'm an embedded platform engineer at Cisco Meraki and we are a cloud-based networking company based out of San Francisco. And I'm here to talk to you about migrating to Yachto and how you can migrate your build system over to Yachto. We as a company build a number of products that we sell, ranging from wireless access points to security appliances, layer two and layer three switches and security cameras. These products are built on a number of different hardware ranging from different CPU architectures to different SOC vendors that we use and a number of auxiliary chips like LTE modules and switching hardware. And we use a number of different kernels for these products ranging from 3.4 all the way up to 4.9. And for those of you wondering, these are actually the code names that Linux uses for their releases. Who comes up with these? Anyway, all these products are built out of a single repository we call router. This repository contains the main entry point, which is a make file to do all the builds. And this make file includes other product-specific make files and open-work make files and so on. This repository also contains all our kernel checkouts and all our user space apps and drivers and SDKs. Anything that goes into a build lives in this one large git repository. This legacy build system is built around an old checkout of open-work. And we have the single make file that I showed orchestrating the whole build. This make file is handwritten and that's never a good idea to do. It has 13 years of craft in it with people adding in all their changes however they want. And a typical build looks something like this. You specify the product you want to build for and you invoke make in that directory with that make file and it starts a build. So you might be wondering why am I here talking about Yachto? Why don't we just take the build system we have, improve it a bit, upgrade it and we'll be there, right? Well, to give an idea our open-work checkout is from 2006. And in the 13 years since then we have made a number of changes to this checkout of open-work. At the same time upstream Yachto has upstream open-work has continued to make their own changes to their own checkouts and we have diverged further and further from what upstream does. So at this point it's fairly difficult to pull in the new stuff they have been doing to pull in the improvements, fixes and so on into our build system if not impossible. Secondly, our legacy build system is very painful to use. Builds are super slow, like four hours for a clean build. And that's because of fun declared dependencies which means we have to build linearly. If you do not know whether package A can be built before package B, you cannot sort of penalize them for fears of race conditions and so on. And when we do a git pull it often breaks our builds because we might have left over artifacts in the source track trees and a package you depend on might not get rebuilt and so on. And this git clean thing is a real problem. Here are all the examples that I found in Slack where people have been suggesting git cleaning over and over. Here is me suggesting it. Here is my friend. Everyone suggests this every other day. All the way back to 2015 when we first started using Slack. Not only that, we also had an opportunity. We were working on a new product and because it's a new product we are going to get a lot of the testing for free. We are going to be testing things that are not even covered by automated tests, for example. And the vendor we were working with had good support on Yachto. So to summarize, we have a build system that's pretty unmaintainable at this point and it's slow. And we need something that's fast, more flexible, able to handle our complex use cases. So we thought, why not give Yachto a shot? It seems like a modern build system built on something called BitPake. And it seems like it's better equipped to handle our complex use cases. And it seems to have better vendor support. So we decided to do our first BitPake build. And we followed the Yachto Quick Start Guide to first check out some repositories that they suggest in the Quick Start Guide and then run these commands to do a build and run the image on KMU. And this only took about 10 minutes. And we already had an image that was able to get to the internet and run in KMU. And that was pretty nice. But we wanted to build our own image that is able to run on our own hardware, the new product that I was talking about. So we went again, went to the guides, which are pretty helpful, by the way, and decided the first thing we needed to do was add a new layer and a new distro and a new machine that maps to the hardware we were actually using for the new product. We then created a very minimal package group, which is just a collection of different packages that would help us get to the internet. And then we create an image that simply installs this group of packages and we get BitPaking again. This again just only takes about 10 minutes. And we already had something that was booting on real hardware. And this only took like a few days of work. And that was pretty nice to see that it's so easy and nice to work with. But, of course, it doesn't look much like our Meraki products, right? It doesn't have our proprietary stuff. It doesn't have our packet processing engine, all our packages. And the packages that are there might be configured differently, might be built differently. So they don't exactly work for us. So we decided the next thing to do is to port packages from our existing open world-based build system to our new build system. And what do I mean by that? Well, there are several packages that are missing in upstream Yachto. And that's because they might be proprietary, or maybe the community doesn't use them as often. And we also want to make sure that the recipes that are there are modified to our needs. And while we're in the migratory path, we also want to make sure that the delta between what our existing build system does and what our new build system does is kept to a minimum to avoid like confusing errors where it built from one, does something different or so on. But before I go into that, I want to give a quick crash course on BitPake for those of you who are not familiar with it. Starting with recipes. A recipe is a basic unit of input into a BitPake run. And it's a set of instructions that are passed by BitPake and then executed to produce something we call a package. And a recipe is written in the form of a BitPake file where it's something like the package name followed by an underscore followed by the version. And this version is passed into a variable called PV, short for package version. And that will be relevant in a little bit. We also have the concept of appends. And appends lets you modify an existing recipe without having to modify the existing recipe's BitPake file. That lets you keep your changes separate from upstream changes. And that's obviously very useful when you're trying to maintain things because you don't want to keep modifying upstream because mergers become a problem. Appends can be wildcarded. The second example where there's a percentage sign, that means that append applies to any version of the code recipe. They can also be version specific so the append only applies to that specific version of the recipe. And finally, we have layers. Layers let you sort of group recipes and modifications and classes into separate groups in a separate directory. So you can keep them contained and sort of define a logical group around them. So back to porting packages from open world to Yachto. What do I mean by that? Well, we want to make sure that the same sources are used, the same package versions are used. We also want to make sure the same patches are applied and the packages are configured the same way and compiled the same way. So what does that look like? Well, we have on the left the original recipe provided by upstream Yachto at a specific version, let's say 1.4.31. And what we do as the first step to modify this existing recipe is to create a BB append in our own layer. And then we set BV to this new value. I mentioned earlier that BV is passed out of the filename, but you can actually override it in an append. And even though the append is wildcarded, it's sort of like changing the version number. It seems a bit hacky, but it works for us. And we're kind of okay with it. We also have to update the checksums because we're using different sources. We want to make sure our Meraki specific patches are applied or any other patches we might have, like security fixes and so on. We want to make sure the same configure options are used and the same compiler flags are used. And this seems fairly straightforward, but not all the packages were this straightforward to port. Some were actually fairly tricky, requiring several patches to the upstream sources to make them work for our thing. We would enable some configure flag, and now we have to patch the thing to make it work in the Yachter thing. And some of the packages were actually missing, like I mentioned. And we had a lot of these packages to port, about 50 of them. And we just didn't have enough time because, as I mentioned earlier, we're working on bringing up a new product that we had to launch at a given deadline and so on. So we decided to recruit help. We organized the Yachter porting day. We created a jurapic, added all the recipes that needed porting to it. And we did some prep work, like making sure all the recipes are labeled easy, medium, or hard. So when people are picking, they can decide what level of comfort they're at with it. We also asked everyone to do a build the day before to speed up the build on the day off. And we sent our invites with promise of snacks and boba. That always works. On the day off, and actually, this is the highest resolution image I could find out of the recording. So excuse the potato recording, but on the day off, I gave everyone a tutorial of BitPake and showed them how to port a recipe. And I asked everyone to go and assign themselves a recipe to port so that we don't have multiple people working on the same recipe. We also asked everyone to use Slack so that if someone has seen a problem and solved it, maybe others can benefit from it too, rather than having to ask me and a couple of other people who need Yachter. And this turned out to be a huge success. We started at like around noon, and by the end of the day, we had 35 recipes ported out of the 50. And a lot of the difficult ones were ported too, which was pretty impressive. But I think most importantly, it helped get people on board. It helped them get a taste of Yachter and how it's fast and easy to work with. And that was really helpful in the long run. So as a tip, if you're working on some similar migration, doesn't have to be the build system, get people on board early, address their concerns, and it will also help you achieve your objectives faster, like it did ours. Also want to talk about how we extended Yachter for our needs at Meraki. Yachter might not have everything you want for exactly the use cases you have, but Yachter is super flexible. It lets you build things on top and it provides all the building blocks to build those features that you want for your specific use cases. Our specific use cases were building out of disk because we had that one large repository that people check out and build out of and make modifications to. And also enabling features and disabling features for a on a per product basis. And these are the two I'll go into, but there were several other things we added as well. So starting from building from disk, the first thing we tried was something called external source. To use it, you simply inherit this class, which basically gives you all the functionality implemented by this feature. And you set this variable to point to your sources. The problem, however, is this class makes it directly point to the sources on disk rather than copying the sources over into the Yachter staging directory. And what that does is if you have a broken package that, for example, creates artifacts in the source directory, you end up polluting your whole repository. And then you have to go clean again to produce a clean build. And then we're back to the get clean problem, which we really didn't want. It also cannot handle multiple sources, which we had to do for a few packages. And if anything changes, so let's say you change a C file, it will go configure again and compile again and so on. And that also isn't great. So to address the multiple source problem, we tried using source URI directly. Source URI lets you specify all the sources that go into your build, including patches. So we tried this. We tried the file type source URI, which lets you specify a path on local disk. But the annoying thing is we have to set S ourselves. It copies things over, but it doesn't change S to point to this new path. So it handles multiple sources, but there's this annoying thing of having to set S manually, which is where it builds out of. And it also doesn't address the problem of rebuilds if any of the sources change. So we wrote something we call morocchi source. It's very loosely based on external source. And to use it, you simply inherit morocchi source and you set morocchi sources to point to all the directories you want copied over. But more interestingly, you can specify patterns of files which this task tracks. So what you can do is say, hey, my configure task should only be rerun if any of the auto make files or any of the C make files change. Similarly for compilation, if any of the C and C++ files change, do rebuild. So if let's say you're working with a package, you only change a C file, you don't need to reconfigure. You can just go straight to compilation again unless any of the auto make files change or any of the make files change or something like that. So this clearly handles multiple sources because you can define as many sources as you want in morocchi sources. You can also, this also copy source is over. So it doesn't pollute the source track trace. And steps are only rerun if the set of files that are tracking changes. And this is all built using fundamental primitives that Yachto provides. We didn't build estate caching or file check sums or prefunks. These are all cool concepts provided by Yachto for you to build on top. And we should all take advantage of it if we have specific use cases. So that helped us do nice clean external form disk builds. And now I want to talk about picking and choosing features based on a specific product. Some of you might be familiar with machine features and distro features. And what that lets you do is say, hey, for my product or my distro or this image, I want PPP enabled or I want Bluetooth enabled or something like that. But of course our features were much different. We wanted, for example, LEDs enabled or we wanted the LTE drivers installed or something like that. So they didn't quite map. So we took inspiration from it and we wrote something called morocchi features. It's simply a global variable and you set to it in your machine config or wherever you want to set the value of it. And it lets you, for example, pick which packages you want to install or configure a package differently or something like that. And the way you do that is by setting the features you want in your machine configuration. So, for example, we want the bird feature. And in your package group you can simply say, hey, if this bird feature is in morocchi features, install all the related dependencies. And we can do all sorts of cool things with it just like machine features and distro features. So far I've talked about all the things, all the places where Yachto really shines. Extensibility, flexibility, speed. But I also want to talk a bit more about room for improvement in Yachto. Of course there's always room for improvement, but I've picked two things I want to talk about specifically. And the first one is the problem of mismatched C library headers. Some of you might have seen this recipe called Linux Lib C headers. If you have ever wondered how the C library itself gets compiled, it starts with kernel sources itself. You essentially run make headers install on your kernel sources for a specific version, let's say kernel 4.1, Linux kernel 4.1. It spits out a bunch of headers that you then use for Lib C's own build. Then the C library gets built, spits out its own headers which are then used for an actual application that you might want to run on an actual machine. Our problem was we were building a different kernel, Linux 3.1.4, which was running on a machine and we wanted to run ETH tool on this specific machine. But the problem is ETH tool is one of those applications that talks directly to the kernel via the netlink API. There are several other such user space applications that need to know exactly how to talk to the kernel over iOcto's or anything like that. But this is currently not possible. And that is because of this disjoint view of build time and run time environment that it has. At build time it sees a header saying this is the API to talk to the kernel. At run time it tries to use it and maybe it's not the same. And the way I ran into it was I was doing my usual builds and this package started running on a machine. And there was this constant speed underscore 5,000, now undefined. So I did some digging through the Git history and I saw that a colleague of mine had made this change. She had added the use of speed 5,000 into the program, into the C++ file. So I did some more digging, see where it's defined and turns out she had actually patched the kernel. But this is all legit. She had actually patched the kernel 4 into 3.14. And this is totally legitimate. Even upstream kernel has this patch. So we simply are back porting the patch. But again, we just weren't able to compile because we wrote an application that was meant to run against this specific set of kernels that were either patched or were new enough to have this patch in it. But we just couldn't because of this disjoint view of the kernel. Our solution was to write our own recipe of Linux Lipsy headers. I know there's a big fat like 50 line comment in the open source Yachto Linux Lipsy headers recipe saying you must use this recipe and nothing else. But I think it's incorrect and we can talk about it if you want to talk about it later. So what we do is we write our own recipe which always matches the kernel and that way the compile time and run time view of the world is coherent. It's built against the exact same headers that the kernel is going to be built against. And we did this again like I said by writing our own Linux Lipsy headers recipe. On the left we have a version recipe at 4.1 let's say and it fetches those exact sources of the kernel. What we do is set pv ourselves and it always points to the version that the actual kernel being built is built out of. We also change s to point to the exact sources that the kernel is using itself. That way we know that there's literally no difference in the headers or in anything else that might go into the build. And the rest of the recipe is fairly similar except like a few minor caveats but the rest of the recipe is fairly similar. And this solved our problem even though it's a little bit hacky. So the lesson we learned from this exercise was some hacks are okay if they help you move forward and unblock here. But of course in the long run we would love to work with the community to actually find a middle ground where the community is happy other users of this recipe are happy and we are happy. I also want to talk a bit more about layers and how they can be so much more than they are today. I touched on BB layers earlier and how you can pick and choose layers using that variable. But what I did not mention is that layers cannot actually be programmatically selected. They're sort of statically built into this one file and I'll go into that a little bit. So let's say you have your open source yachter provided layers that you use if you're doing a build. Pretty much that's these layers. Then we have our Meraki specific layer. This contains all our modifications, all our recipes. And let's say we start a build for our particular machine. What we would have liked is for some way to conditionally programmatically insert a new layer for this specific machine. That way we could keep all the changes and modifications for this one machine in its own layer like keep it segregated, keep it clean. But as we know it's layers are selected through a variable called BB layers. And this lives inside a file called BBLayers.conf which lives inside the build directory. So every time you create a new build directory this file is sort of statically put in there and if you want to choose different set of layers you have to go modify this file. You cannot sort of like conditionally select BB layers because that's the thing we were really missing. Sure we could have written tooling around it but that seemed like even more hacky than it's worth and we talked to some people in the community and they seem to have run into problems trying to do that. So we stayed away from it. Similarly if you want to do a build for a different machine we would have liked a new set of layers to be picked for that particular build. But again this is not possible because you have to go modify this file before you can start the build. So if you're working with two machines you have to kind of go back and forth and take Meraki for example we're working with like 10 machines 15 machines and that becomes a real headache. But layers are still very useful. Everyone should use them as long as you don't have to switch them in and out often enough. So let's say we have our same set of layers. We use layers for something we call back ports layers. So we have our own set of layers the same set and the open source layers are checked out at version 2.5. And let's say they contain a recipe for curl at version 1.4.1 but we find a security vulnerability in 1.4.1 and we have to upgrade it as soon as possible to address the security vulnerability. So we decide we need to upgrade to 1.5 and we see that upstream Yachter the next version perhaps already has a recipe for 1.5. But we don't want to just blindly download that recipe and add it to our own layer because we'll end up with like these tons of recipes that we'll have duplicates for once we go to the next version anyway. So what we do is we create a new layer called the back ports 2.6 layer and this layer contains all the back ports from 2.6. That way the changes are kept in their own thing and we know where they came from. They came from the next version of Yachter. At some point of course we're going to upgrade our own Yachter to 2.6 and now we have two copies of this recipe but this is all totally fine because we can just simply get rid of the back ports layer and we have this single coherent view of the world again and it's basically a no op to remove that layer at that point and that sort of gives you a nice clean path to upgrade and your own checkout of Yachter without having to stick to the upstream releases and you can upgrade at your own cadence, you can decide when to upgrade and so on. All of this work that went into Yachter and all the people who helped with it led to a number of successes. The biggest one was of course the build times and this is a super exaggerated diagram or a chart but it actually gave us a 16x speed up over the existing build system. We went from literally 4 hours to 15 minutes and that's a huge gain in developer time. We no longer have to start a build and just go home, you know. So that's the real win for us. We save a lot of developer time, we save a lot of frustration around like waiting for builds and then if it fails you have to clean and start all over again. Most of our products are also now on Yachter and that means we gain more and more benefit from it as more and more products move to Yachter at Meraki. You might see this like sort of this jumps in the number of products ported and that's because let's say you have a certain class of products let's say our wireless access points. The first one might take a long time to port because you have to test it and think about all the base but after that all the other wireless access points are very similar to each other. So the next few just take you know like a week to port two or three at a time. So that's how we get this like sort of and that's an important thing to remember like if you're porting the first one will be difficult it will take a lot of time but the ones after will sort of come for free. And finally working with Yachter has been a pleasure it's super easy to maintain, super easy to upgrade and we don't have to write recipes often times by the time we need a recipe upstream Yachter already has a recipe for it and if we do write our recipes first upstream Yachter we like open sourcing them so that the community can take advantage of it. And I want to close with a few thoughts. I really can't stress enough how great Yachter has been to work with the community has really put a lot of thought into designing it making it flexible sort of giving us a general purpose build system and not just specific to one use case and that has helped us build a lot of features on top of Yachter and we would love to open source all those features there's nothing proprietary and we have taken a lot from the Yachter community and we would love to give back and while I'm here I also would like to propose a more of a modern RFC process rather than perhaps using mailing lists I'm not the biggest fan of mailing lists and something like what the Rust community uses like GitHub or GitLab or any other modern tool or platform I think it would really help drive input into new features that get design and get more participation from the corporate and the open source and the community and so on I also really want to stress how important it is to get people on board early it will help you achieve your objectives faster like I mentioned but it also helps reduce friction listen to what the people have to say in different sense show them that you care what they think otherwise by the time you introduce the change there will be a lot of friction and pushback and it's too much to deal with at once I want to close by thanking Ram who was the other main person who worked with me on Yachter to make it the reality it is at Meraki today and Matt Prokos at Meraki who did the initial proof of concept and helped us realize we can actually do it at Meraki of course everyone will help me prepare the presentation and the amazing Yachter community over the last year or so I have had a bunch of dumb questions to ask on the IRC and always met with good useful responses that I have been able to work with and also I want to thank everyone at the Yachter Dev Days last year who also answered a bunch of our dumb questions and helped us think through some of the problems we were facing and decided to present here this year if you want to reach me that's my email address you can also come find me if you see me and ask me any questions you might have and if anyone has questions I can take them now so using core image install also to decide which packages go into the thing right we did think about that but that feature specifically addresses what things get installed a lot of the times let's say we enable some feature we have to configure a number of packages differently we have to compile them differently we have to do all sorts of other things besides just installing a different set of packages so that's why we decided to use that good question yeah right so we did look into BSP layers quite a bit and that's exactly sort of the thing we want to do but I wasn't aware that they were dynamically selectable do you still have to go edit bblays.com to do that right so we looked into that the second option you mentioned but we sort of decided not to do that because we talked to some people at the yachter depth this last year and they had done something similar but they want huge fans of it so I would love to hear more about it and maybe we can actually fix that issue okay cool thank you I'll definitely look into that other questions so the new product we started working on was towards the end of 2017 we launched in 2018 and from then on we started porting all our existing products and today we are at 60-70% done right we were like we're going to launch the new product on yachter exclusively and then move everything over we haven't been working on it like full time but as we get time and resources we try to port them nothing really except resources and time we would we would love to start as soon as possible there's like we could make the Maraki source generic and let the community use it and stuff like that so definitely we want to contribute that's not in question but just you know the timing and resources to do it right so do you mean to ask if we were starting from scratch and we didn't already have existing products there's a tough question to ask or to answer rather because I mean if you have a very simple use case maybe build root would be fine for you if you don't like make files then maybe build root is not fine for you depends I think as you scale if you predict your number of products going up and your use case is kind of specializing go with something that's more you know complex and helps you deal with the complexity if you just only have one product and you are doing minimal modifications to upstream then and anything works really they're both equivalent in terms of you know build times and features and so on but just how flexible they are and whether you like make files I really don't like make files anything else thank you very much