 OK, can you hear me? Good. So thank you for coming to my talk on migrating a production platform to Yachto. My name is Mitch Gaines. So about 12 months ago, my boss came out to me, and he said, and I'm paraphrasing, he said, Mitch, we're not able to source the hardware we've been utilizing for our Edge devices anymore. And this might be a similar story for a lot of you, given the supply chain issues that we've all been facing. So the task that he was asking me to do was make our system run on an ambiguous set of hardware, really whatever we could get our hands on. Is it working? Good. So it was really whatever we could get our hands on. At the time, our fleet of Edge devices was running a Debian-based platform. And on top of that, we had a pretty mature application stack with a bunch of tooling built up around building, testing, deploying, or software. And if your system's or anything like mine, you probably feel like it takes 100 steps to do a seemingly simple task, like updating a package on your fleet of devices. Similarly, if you have a bunch of applications on your application layer that are all communicating with each other, one thing that might keep you up at night is not knowing whether or not the package that you're about to deploy to your Edge devices is going to be fully cross compatible with all the different versions of the applications that could be running on your system. Now, I'd heard about the Yachto project, but I hadn't used it extensively. But it seemed like the solution to a lot of our problems. So first, it allowed us to build our system for different hardware architectures so we could support our old boards and our new boards. Second, it allowed us to do full system upgrades and integrate that pretty seamlessly into our workflow, whether that be over the error upgrades or delta upgrades. And third, it significantly reduced the constraints on the permissible boards that we could use. It's such a lightweight system that we were able to reduce the hardware constraints that we had. Now, a few years ago at this conference, Chris Simmons, who some of you might be familiar with, gave a really insightful talk on Debian versus Yachto. Which one should I use for my project? And the biggest takeaway from that talk for me was Debian is great if you're building an off the cuff system for one off device maybe that you're prototyping with. You don't exactly know what packages need to be on it. And Yachto is great for the production edge devices where you want all of your devices running the exact same thing. So it was obvious to me and my team that Yachto had a lot of benefits over Debian. And that was kind of the route that we decided to pursue. What I had no understanding of was what it would take to migrate our entire platform with almost a decade's worth of development behind it over to the Yachto approach. So my name is Mitch Gaines. I'm an embedded systems engineer at a couple startups. And among other things, I build Yachto-based systems for these companies. Primarily my work right now is at MBU where we do building automation for multifamily apartment buildings. Previously, I worked in customer success, so that has kind of influenced my career to building robust systems. I've been on the other side of this before, so I really like building fault-tolerant systems that fail cleanly. By the end of this presentation, I'm going to give you clarity on what the path of migrating your system from Debian to Yachto is like. If you're a manager, you'll have an understanding of the amount of resources required to make this migration. And if you're an engineer, you'll learn a few techniques for efficiently making this migration based on my experience. And should you choose to make this migration, my goal is to give you some ways to approach it without skipping steps to get to a Yachto system seemingly quicker. So we'll talk about a couple things. So first is why I made the migration the way I did. And second is what tools do I have to assist me in making this migration? When I talk about why I made the migration the way I did, we'll compare and contrast the Debian and Yachto build processes. And we'll draw a lot of comparisons between the two and talk about how you can utilize your Debian build process to seamlessly integrate you into the Yachto build process. And we'll address a number of shortcut paths to apparent success that you might run into that maybe you shouldn't pursue. And when I talk about some tools I have to ensure my success at making this migration, I'll talk about ways to exercise the layer-based architecture that Yachto utilizes, as well as ways to integrate Yachto into your application developer's lives. Because as soon as you change all their tools on them, you know they're going to kick and scream. So I'll talk about ways to integrate that pretty seamlessly. So when we're comparing our Debian and our Yachto build processes, what's important to understand is Yachto piggybacks off of a lot of technologies that already exist. So your Debian build process is part of Yachto. So I believe it defaults to building IPK packages. But you can configure it to build Debian packages as well. The difference between our two build systems, though, is that Yachto, the output of it, is a full system image. And baked into that system image are my applications and my whole application layer. Whereas with Debian, in order to get my application layer onto my system, I'm going to flash my system onto the board I'm going to use. And then I have some agent that pulls down my application layer using a package manager. So the steps are pretty disjointed from one another compared to the Yachto approach. Now I'm picking on Debian a lot. But really what I'm talking about can be applied to any generic Linux distribution that you're migrating from to Yachto. So we'll take a deeper look at the Debian build process. Now I'm sure a lot of you might be intimately familiar with this. So we'll quickly go through it. But I want to draw some comparisons between us and the Yachto build process. So before we go through this, first we need to flash our operating system onto our device. So we have a system running a Debian platform. And for each application, I'm going to go through this process to get my application layer to Debian. So first, obviously, I'll pull the source. And whether that's from a Git repository or a developer machine, I get the source. And with that in hand, I can then cross-compile my source for my target hardware architecture. Now what's important here is if I need to support multiple different hardware architectures, I've got to do this step more than once. So I end up with multiple different sets of binaries that I can then take and build Debian packages with. Now I build those Debian packages using some tooling that permits me to do that alongside whatever dependent files I have for my system, some be a config file or a service file, that sort of thing. And with that Debian package in hand, I then upload it to some server, the package management server. And my devices have an agent running on them, whether that be Ansible or Puppet, that will pull my changes and I update the system. Now compared to the Yachto build process, this is a big complicated graph that I took right out of the Yachto manual. But in order to utilize the Yachto build process, I run Bitbake. And Bitbake is the tool that will build my system image. And it does all these sets for me. So the first thing it does is it pulls all of the source down for my applications and third-party applications alike. Does whatever patching operations may or may not need to happen. And then it cross-compiles all of my applications. So you can change which target you're cross-compiling for by passing in a different machine configuration file. And you can switch between those by changing a Yachto environment variable. So you cross-compile for several different targets, whatever target you want to utilize, and generates a bunch of packages for your system. And with those packages, it generates your root file system and basically generates an image that you can then flash directly under your devices. And if you want to utilize an update mechanism, such as if you're doing over-the-air updates, you can also generate kind of a bundle that will upgrade systems and that sort of thing. So one of the things you might be thinking, right, if you're running Debian on your systems right now, is Yachto seems great. There seems to be a lot of benefits, but the upfront cost might be too high for you. You might think it's too high for you. And yes, supporting a different machine is a little bit cumbersome, but it's not that bad. Really, it's a question of short-term gains or long-term gains, right? So making the migration to Yachto has a bigger upfront cost than supporting a different machine in your build process. But the long-term gain is definitely there for migrating to Yachto. Now, if you're utilizing Debian in your build process, you're actually at a huge advantage to migrating to Yachto. That's because you have an exact blueprint for exactly how your system needs to build and install on the system. Now, in the Yachto project, there's a list of recipes. So each application generally has one recipe associated with it. When I run BitBake, it'll parse out the recipes and generate a bunch of tasks that run within BitBake. And those tasks can be things such as fetching my source, compiling my source, and installing it. There's a long list of tasks that it creates. After making this migration, there's going to be three tasks that you're going to become intimately familiar with. So the first is our do-configure task. Second is our do-compile task. And the third is our do-install task. Now, with our do-configure task, I have managing patch files here. That might raise a few eyebrows, because if you're familiar with the Yachto project, you know that it has its own patching mechanism. What I found myself doing, though, and a problem that I ran into was utilizing the Yachto patching mechanism requires me to put my patch files into the Yachto environment. That being said, I also need to support my Debian build system, because for some undefined amount of time, I'm going to need to support my Debian devices that are in the field alongside my new generation Yachto devices. So I can't break things in Debian for the sake of getting my Yachto builds to run. So I had these patch files that lived in my source control. And they lived in my source control because they were patching third-party SDKs that my applications required. So in order to get them into Yachto, the bad approach would be to create two copies of the same patch files. So I wanted to utilize one set of patch files in both of my systems. So what I found myself doing was utilizing quilts and executing quilts on the host in order to apply my patch files within Yachto. And I did that in my do-configure task. And the cleanest way I found to do that would be to create a task in my make file and then execute that task whenever I run my do-configure task. So I can do like, oh, we run make, and then my patching task in my make file. And I found that to be the most effective way to maintain my patch file sets. Now, similarly, once we move on to our do-compile task, I can't change things in my make file that could potentially break my Debian builds. So this would be a good chance to utilize the Yachto patching mechanism. So create a patch file for whatever make file changes you're making and have that live in the Yachto environment. And that'll maintain the separate build systems for you. But more fundamentally, in the do-compile task, I actually found this to be the hardest step of the entire process for all of my applications, getting them compiling in the Yachto environment. And one of the reasons for that is because your application developers develop the applications. And in the make files, they may set a bunch of environment variables for the cross compilation that they don't realize are actually machine specific. So a big problem I ran into was Yachto would go apply a bunch of variables to my GCC command, and they would actually be in conflict with one another. So you're going to want to really have a good understanding of what all these variables that are set are doing in your GCC commands. And I found that I needed to get help from my application developers for this. Because frankly, you've got a lot of applications. You're not going to be intimately familiar with every single application on your system. So utilize the resources that you have to get help on that. But also, Yachto is about to become part of your application developers' lives. And it's good to start introducing them to the topic of utilizing Yachto as their build system. Because if you introduce it aggressively and all at once, it might be a bit of a problem. So this was a good opportunity to get my application developers acquainted with Yachto. And then the last problem I ran into with my do compile task was I worked really hard to get my application compiling in the Yachto environment. And I got it compiling. And I said, great, I'm done. And move on to the next task. But the problem I ran into was two weeks later, I try and actually execute it on the system. And it's hitting all sorts of issues. It's segfaulting immediately. So you're going to want to test runtime behavior alongside while you're debugging your compilation task. So you're going to move into your do install task. And you need to test the behavior here in order to execute it on your system. The do install task is super, super easy, though. If you have your Debian packages, inside of that Debian package is an install file. And that install file defines exactly how you need to install the binaries and the config files and service files, exactly where it needs to go on the system. So utilize that tool that you have with your Debian build process to use it in your install task. Now, as we've kind of gone through this and compared our Debian and our Yachto build processes, one of the things that you might be thinking, one of the things that I was thinking when I was going through this process is how can I incrementally get to Yachto instead of taking all of my Debian stuff and throwing it out and then building up this new thing, how can I incrementally get there? There's a few ways you might be thinking about that. First is I see that Debian, or I see that Yachto utilizes Debian packages as part of its build process. Can I just inject my Debian packages somehow into Yachto? And maybe I was naively thinking this. The other thing you might be thinking is maybe I can containerize my application layer and have it run on my system. Then I don't have to touch any of the Yachto build system. So to address the first point of injecting our Debian packages, there's a couple of ways you might be able to do this. One is exercising a package manager. Yachto allows you to put a package manager on your system if you want. The problem with that, you can do that if you want. The problem with it though is you're separating your application layer from your system layer and what have you really gained by moving to the Yachto build system? It might be a good incremental step, but it's by no means the final solution in my opinion. Now, the other approach you might be thinking of, and I thought I was clever when I tried to do this myself, was I know my recipes can unpack a Debian package inside of them, so maybe I can just get my Debian packages into my Yachto environment utilizing my existing Debian platform that I had built tooling around and bypass the whole need to do this recipe business. The problem with that lies in dependency trees. So for all of your applications, they have dependencies and they have tertiary dependencies and dependencies beyond that. And in order to utilize that effectively, you're going to need to map out the dependency tree and get all of the dependent applications into your build system. So this quickly became a little bit overwhelming and I deemed it not usable. Now, in terms of containerizing our application layer, again, we run into the same problem by separating our application layer from our system layer. Without a lot of discipline, we bypass our ability to do a full system integration testing with the whole application layer and the system. And again, we can push packages without, you know, and have a bunch of different versions running on different devices. And in addition to that, I actually I found one of the problems I ran into was that containerization software like Docker is super heavyweight on the system. And it ends up being a pretty big resource hog for your system. So I found that it added a lot of bloat to my system and I didn't want that. So I deemed that, you know, not feasible for my project. So, I mean, the great thing about Yachto is, you know, if you need containers for your applications, you can put containers on there. If you want a package manager, you can put a package manager on there. It's really, really flexible to suit your needs. I would really recommend utilizing the Yachto build process though, because, you know, among the other reasons I've provided, it provided a much simpler provisioning process for my devices, right? I now have a system image that has my application layer baked into it, which allows me to flash that directly to my devices. And I basically just have a setup script that runs on first boot and self-destructs that sets up, you know, keys and certs on my device and then never has to run again. Super, super streamlined process. Now, for supporting multiple different types of devices, right? So, Yachto has a layered-based architecture. So at the base layer, I have, you know, my pokey layer and my open-embedded layers. And above that, my board support layer is my BSP layers. So that could be like, you know, Raspberry Pi layers or Sunsea. I might also have a custom board layer that I make. And on top of that, I have my application layers, so third-party applications and my team's applications. Now, for supporting different boards, though, you know, say I have a Raspberry Pi and, you know, Sunsea board, in order to build my different targets, I'm going to swap out those layers as I need them. But inside of your custom layer, right, you might have different configuration needs for different boards. So you can specify the machines that you want your configurations to apply to for the specific build that you're doing. So it makes it super, super easy to support different boards within your system. You end up with two different image files that you can then divvy up for the respective boards. So, how do I get from Debian to Yachto? Okay, so how do I build up this Yachto system utilizing what I have? So the first step in this process is make your system boot, okay? Figure out the hardware requirements that you have and make your system boot. Yachto also provides, you know, LTS releases, so figure out the Yachto release that you're going to be on, determine your kernel and boot loader requirements. But also at this point in time, you should start considering the update mechanism you want to utilize. The reason for this is, at least for me, I was using Ralk and had an over-the-air update utility and an AB partitioning scheme. And I found myself doing a lot of development at the boot loader layer for writing scripts, you know, deciding which partition to use. You don't need to have the end-all solution to this just yet, but you need to start thinking about it and make a system that boots. Now, when I approached this, I wanted to rush to the end and get my applications building in the Yachto environment right off the bat. The problem I ran into is I got it building relatively easy and got one of my applications building relatively easily in the Yachto environment. But then I figured out the hardware requirements that I had and then all the cross-compilation stuff changed. So you want a system that boots that you can, to know your hardware requirements so that you can build your applications around that. Next, you're going to want to integrate your application layer. So once you have the system that boots, integrate your application layer. Call upon support from your application developers and get their expertise in knowing what system libraries you need for the applications and utilize your current Debian build process as your blueprint for getting you to Yachto. After you've integrated your application layer, you need to migrate your application developers. It's a lot more challenging than it sounds. Your developers aren't going to like change. But if you've integrated them into the process early on, hopefully this won't come as too much of a shock to them. A couple of tools that I found really helped migrate my application developers over to the Yachto approach. First was DevTool. DevTool is a really, really great tool. In fact, this is probably one of the big tools that ultimately sold all of my application developers on the use of Yachto. DevTool made it really, really easy to do application development on my devices, on the edge devices, because for all the software that my application developers were building locally and needed testing, within seconds they could get their compiled binaries onto a single development device and test it and then cleanly take it off and no longer have it on the system. So this made it really, really easy for them to use. Now the second tool, it's a pretty basic tool in Yachto, but it's called Autorev. One of the problems that I ran into was when I first started this is I had a bunch of applications that you set up in your recipe the repository that you want the source to pull from. And within that you have to set a commit hash that your project is utilizing. However, when you have a bunch of projects, it quickly becomes a little bit cumbersome to update the commit hashes to incorporate changes on a regular basis. One of the problems I ran into was an application developer pushed a change that broke my Yachto build for their application and I didn't recognize it until a month later. And you go back to them and say what did you change? And they've got no idea, right? So what you can do is instead of utilizing the commit hashes, you basically set the commit hash to the Yachto environment variable Autorev and it will auto revision all of your applications and it'll help you to stay on top of changes that are happening in your system. And in addition to this, I manually run nightly builds. I say nightly, but really it's more like every other night or maybe week way sometimes. But I stay on top of the changes that are happening in my repositories by frequently running builds and making sure my developers didn't break anything so that we can stay on top of changes that are happening and they don't break anything. So with all these techniques you can take your system from Debian to Yachto. I encourage you to make this migration if requirements arise that require you to support different machines in your production edge device setting like it did for us. Now, feel free to connect with me. I send me an email if you have any questions and at this point I think we have a few minutes for questions so I'll be happy to answer anything that you guys might have. Say it one more time. So, okay, so the documentation I'm pointing to is specifically for the Dunfel release which is the release that I'm utilizing for my system. I think you can change the URL to be the current release if you'd like to. Yeah, latest. Okay, so if you wanna utilize the documentation I highly recommend you're gonna be very, very well acquainted with the documentation by the end of this migration. Go to docs.yachtoproject.org. Don't use the link in my presentation. That's the link that I look at but it may not be the same for all of you. Good catch. So, it took me about eight or nine months and that was me as a solo developer on making Yachto. I will say a lot of the time sink was in me not knowing how Yachto worked at all. I was vaguely familiar with it prior to this project but this project really leveled up my amount of knowledge in the area. So I made a lot of mistakes that the point of this presentation is to relay some of those mistakes that I made if you're using Debian right now and haven't really used Yachto. So a lot of the time was in just mistakes being made along the way. So, okay, so for those online the question was how excited were my application with all the first to learn Yachto? They weren't excited at all. You know, they wanted to kick and scream when you'd start changing their tooling and frankly they had built up a bunch of tooling around their current approaches that they developed over years and years and years that only made sense to them. So in my opinion the Yachto approaches is a little bit simpler and once they realized that it was not that big of a deal. Any other questions? Yep. Okay, so for those online the question was it seems a lot simpler to make the packages available by building Debian packages compared to Yachto where you're building everything from source every time and then making it available. So the mindset is a little bit different with Yachto. At least with the approach that I made with doing full system upgrades basically instead of pushing a package out to a repository and having my devices pull it down I would generate a system upgrade artifact. So for me it was rock and I could take that artifact and upload that to a server. And then from there I would update a development device, make sure it works, do some testing on it and then deploy that update out to all of my devices. So it's more of a full, it's more of an all-encompassing approach to upgrading as opposed to piece mailing specific packages on all of your devices. Does that make sense? Okay, good. Yep. So we haven't adopted that yet. I'm not opposed to the idea of utilizing it. I haven't quite figured out how to actually use it though. I found it a little bit complicated to set up. So I'm willing to potentially use it at some point but we haven't found a way to really incorporate that into our system yet. Say that one more time. Okay, so the question was do I set autorev for all of my third party recipes as well as my applications? The answer is no. So I only set that for my applications themselves. I don't find that I need to touch the third party recipes virtually at all. If I'm doing that, I know I'm doing something really, really weird and probably wrong. So I stay away from that if I can but just for my applications is where I set autorev. That's a good question. Anything else? Yep. So the question was what do I miss most from doing it with Debian? That's a really loaded question. So I mean, I guess the thing I miss most is my application developers really, really know Debian super well. So it was really easy to call upon support from them if something was breaking in the process. Whereas with Yachto, I'm kind of on my own. So when things break, it's all on me to fix it. So having that support with Debian is nice on the team. Any other questions? Yes. So right now, basically what prompted this whole project was we had historically been using one fleet of hardware, right? But we couldn't source that hardware anymore. So we need to support a new generation of it. And so really we only have two different types of hardware that we are supporting currently. We don't really have a strict timeline on when we're going to end of life our old ones. It's kind of once they start breaking, we'll start replacing them. But we'll probably maintain them as long as we can. Does that answer your question? But the great thing was part of the decision to go with Yachto was we knew that we might end up needing to support more hardware in the future, right? Things are going to keep changing and evolving. And eventually it's a little bit cumbersome to support one set of different hardware. It's extraordinarily cumbersome to support 10 different sets of hardware, right? And foreseeably we could get there. So part of the decision to go with Yachto was because the future is uncertain. We could be switching to something else as soon as supply chain issues make boards cheaper again. Once the supply chain issues work out. Any other questions? Good, well thank you for having me. I hope you got something useful from my talk. And yeah, have a wonderful rest of your day.