 time. All right let's start. So I'm obviously not Ross Burton. If you came here to see Ross I'm sorry he's not here but I am. He has a better accent but I'm way funnier. My name is Beth Flanagan. I am the Release Engineer for the Octa project. I'm also a software developer for Intel OTC. I'm a contributor to the core in the Octa project. Mainly deal with license wrangling, build statistics, GPL compliance tools. I've also written documentation for BitBake for a book called the architecture of open source applications volume two. But what I'm here to talk to you about is my first love which is auto builders. I love auto builders. I started with auto builders in about 2006 working for a feature animation company whose name shall not be spoken. If you buy me drinks I will tell you how someone screwed me out of the movie credit. Then I went on to work for the smart meter or for the local electric company doing smart meter infrastructure auto builders. So if you live in Portland and the electricity goes out, totally my fault. I now am the maintainer of the Octa auto builder project or the Octa project auto builder. So one of the things that I'm going to introduce today is some of the refactoring work that has been done in the Octa auto builder. So there's some caveats here. This is brand new code. Not quite feature complete, but it's totally functional. In other words, it works on my machine. It's stable. In other words, it hasn't crashed on a machine. And really it does need more eyes on it because I'm running into issues where I have no idea why it doesn't work on Saul Wolde's weird configuration that he has. This is literally brand new code within the last two months. So the interfaces and configs to this shouldn't change. They may have something really bad is found. If they do, all of this should be backwards compatible. It's also entirely untested in production environment. We're going to talk about some of the plans that the Octa project has for implementation on this as well as testing. So why other than Ross Burton's not here, do we talk about auto builders at an embedded conference? And really what this comes down is to this whole code compile test loop. You know, an application developer, their code compile test loop is like 15 minutes at max. Maybe if they're like playing dangerous or like 30 minutes, for an operating system engineer, your code compile test loop may be anywhere from 20 minutes, 15 minutes to hours. If you start hacking around with EG libc, and you have to recompile a whole bunch more root F s's, you may be looking at six hours there. So one of the things that's a really nice thing to do is continuous integration when it comes to embedded. Have someone else deal deal with building it and you keep on going on your merry way. The other reason is that auto builders should release software. I'm a software developer, but I'm also a release engineer. And I like doing release engineering, mainly because a few years ago I started doing this because the release documentation sucked. It was horrible. It was all wrong. And really your release process should be in code. Let the auto builder do it. Your release, your documented release process should be push this button. Make your release processes push button. So a little bit of history behind the octo auto builder was originally called the pocket auto builder. Richard Purdy created it 2009. The initial config was about 177 lines had four targets. Somewhere between 2009 2010, Scott Garmin took it over. If you went to Scott to talk yesterday about the miniboard, Scott did excellent work on this. It was 642 lines. 17 targets. Took around 26 hours to build nightly, which wasn't precisely nightly, but it did it only on three machines, two machines. So it was a fairly tight situation. I joined the octo project in November 2010. And then this slowly and rapidly, well slowly and rapidly, depending on how you how you look at it, grew very, very quickly. There were there's been a lot of improvements to the auto builder over the past two and a half years. Some of these have been feature driven. The introduction to OECOR has been one of the major features on shared state auto builder using shared state and having a cluster auto builders all using the same shared state has actually kind of improved our build times. When you have in at one point, we started looking at build times at 42 hours for nightly. The solution to this with some more hardware at it. And one of the gentlemen who worked on this Michael Hall said sitting right out front. So we started seeing a lot of improvements on this. So the current state of things has a lot of good about it. All this work reduced build time significantly. We are now looking at on average six and a half hours of build time to generate a quarter terabyte worth of build artifacts. Images, ABT repos, Eclipse plugin, the build appliance VM. And this doesn't even include like shared state, DL Durr, all those other things. What this allows us to do is do very quick turnaround. We noticed that something got committed to master and it starts breaking things. We can go and fix that quicker than having to sit there and wait 24 hours. And again, this the the current incarnation of the octa auto builder sports a lot of very cool features shared state Durr on an ass. So all the builders can use it, build history, sanity testing, persist DB reuse. All of this got put in and passed to me half years. But the unfortunate thing is it's caused some bad things. One of these bad things has to do with just out of curiosity, who has experience with build bot here other than me? All right, two people. Part of this has to do with how build bot does conditional build steps. If if you're a normal developer, you're used to a conditional being if then else. Build bot like saying, Well, now this conditional build step is run this function if it returns truth and conditions true, run the build step, if it's not run it false. And then what you end up doing is starting to get a very large config file that ends up getting code and config mixed together. And it's icky. It's not so bad for build engineers like me who don't mind hacking through this. But for the rest of you, it sucks. So by late 2020, 2011 or 2012, sorry, the auto builder main config was 2714 lines long. And we started getting into this situation where if I got hit by the lottery bus, won the lottery or got hit by the bus, then we would have a situation where no one can take this over. I started noticing that this was getting to be a problem. But then there was the ugly. This is one of my favorite quotes. I love this book. And you know, I like the picture of Winston Churchill up there criticism may not be agreeable, but is necessary. Phil's same function as pain in the human body. It calls attention to the unhealthy state of things. Prior to one dot three, there was a beta test. And that beta test called attention to the unhealthy state of things. Mainly around people setting up and reusing the auto builder. Other people were having a hard time reusing it. Required advanced knowledge of how things were structured within the auto builder config. And understanding just the concept behind the auto builder. And none of this came as a surprise to me. I've been yelling about it for about nine months. I just didn't have time to fix it. And really what it came down to was a philosophical change that I needed to look at the initial auto builder like a lot of CI systems came out of the need. There was a need to just run this and build, run this and build. And then it moved into this reference design. Well, here's how you set this up. And now it has to move into something that's a usable product for people who utilize the Octo project to actually take and reuse. But the question is how do you do this and give end users rope but not enough rope that they hang themselves with it? And the other question is the Octo auto builder does the one thing that it was designed to do, release the Octo project releases and it does it very well. But how do I do that and enable people, lots of other people to do their things very well? So sometime after 1.3 I sat down and looked at some of the user feedback and I said, all right, these criticisms really have to drive my goals for the next version of this. I have to make it so that people can set up their own auto builders very easy. I wanted to require a very limited knowledge of Python. Nothing very special, just very basic knowledge of Python, very limited knowledge of build bot even. Want to provide good code references and good config references. When the main pain points is code goes one place, configuration stuff goes to an entirely different place, they should never meet. And I wanted to add some very new much beg for features. There are people who have been begging for some of these features like mix and match repos, which I'll go into in a little bit. But there was no really good way to do this in the old architecture without ripping things apart. I also needed to maintain all the old features that I had put in over the past two and a half years. In doing all of this, if people want to do really hard things like have their little lava lamp that turns red if things go fail, I want to enable them to still be able to do those hard things. And in the end, what the new auto builder spits out has to be the same thing that the old auto builder spits out. So the one lesson I learned out of this is the best time to do proof concept is when no one is looking. My manager is sitting right there and he's probably, yeah, he's shaking his head, good. For the first day of Christmas break, I was got bored. And I said, all right, I'm going to start looking at this. So Christmas breaks are like December 19th and I don't think I really slept until like December 24th. Woke up Christmas day, started writing code. By December 26th, we had actually something that started working. Something that could actually spit out images. It wasn't pretty, but it worked. So what we want to do is we want to introduce some of the architectural overview of what we're doing here. I have, this is a contextual joke, if you were at FOSDEM and saw the SystemD talk. SystemD is an init system, SystemD is a platform. The OctoAutoBuilder is a continuous integration system, but it's also an abstraction layer over bit, or build bot. We split out code from core code and build steps. So we have the core auto builder code and then build step code. We also split out config from all that code as well. And this provides an interface to be able to use custom build steps as well as build bot build steps. So there's three types of config files that we're using. First one is autobuilder.conf. Has anyone actually used the old auto builder? Other than me? Oh, good. Autobuilder.conf is basically the global settings file. What this says is where does the auto builder spit out its DL there? Where does it spit out shared state? Do we want to collect build history? Basically all these things that every builder on the auto builder needs to make, or every build set on the auto builder needs to know about. This lives in the config directory that's going to be changed. The main build set or config file for the OctoAutoBuilder is the OctoAB config. Technically that's the only config file you need. That's not a good idea. I'll show you why in a little bit. This also lives in config. What I end up doing is taking build sets, so nightly, nightly x86, nightly arm, and putting them in their own config files. You put them in a config file, you drop it in the config directory, you start up the auto builder, auto builder knows about it. When we look at the main OctoAutoBuilder config file, it has one top level section in that set. That top level section is the only thing that technically needs to exist. You can start the OctoAutoBuilder app with it. It won't build anything, but you can do it. It's the only thing that actually needs to be there. And it only has one property, and that's order. Order dictates on the waterfall page what order do you see things in. This is one of the few places that I've actually modified BuildBot and I'm working on the patch that I did to see if this can get upstream. I am going to show you what the actual file looks like. This is the file that we have right now. And like I said, one property, order. And if you go and look at the OctoAutoBuilder, the one I have running on this local machine, you can see that it's in the exact order. But all the way at the end here, we have these builders that aren't actually listed. No, they are listed. Let's assume that they weren't listed in that order property. Sorry about that. What happens then is that the Octo, that BuildBot, will just revert to its own sort order. This was one of those things because we like seeing Nightly all the way on the left-hand side. It's kind of silly, but we do. The next thing that we have are the config files that actually control build sets. These only have four properties. With four properties, you can run an entire build. Steps, properties, repos, and builders. Only three of these are actually required. That would be steps, repos, and builders. I'm going to take the easy one first, which is builders. Builders just tell you where that build set runs. So if you have a distributed build system, Builder 1, Builder 2, Builder 3, and you want that to only run on certain one of those builders, you just create a list of those builders and assign it that. The repos are a list of repos needed for a particular build set. For example, pocky.get, metaintel, metaqt3, etc. If you look down at the bottom, this is what the repo section looks like. It's fairly straightforward. It needs the name of the repo. It needs the repo URL, BBPriority, and branch. BBPriority is actually at this point optional. In fact, it doesn't work yet. I haven't implemented that yet. You don't even need branch. You can name this hash or tag. I have that in there for convenience because some people can't rock the concept that hash branch, tag, or all the same thing. One of the interesting things here, and I'm going to show you this, is that if we look at Knightly, Knightly has only three repos defined, but way down here, and I'm going to talk about triggering builds in a little bit. Knightly is a worker build set. What Knightly does does is Knightly goes and says run this and then go run all these guys over here. But one of the things that I needed to do, I need to be able to set Knightly's repository, its branch, tag, or hash, as well as every triggered build step that it ends up triggering. And the reason for this is, let's say I want to build master on everything except this person has something in their trip branch that I want to pull in for Knightly multi-lib. I can actually go into Knightly multi-lib and set blocky, contrib, and we'll do sgwmut, because I do sgwmut quite a bit, and it will tell Knightly multi-lib to run that, to ignore everything else. The main reason for this is, and we'll go through this in a later slide, this gets me out of dictating branching policy for people. The other thing that we have are properties. Properties are things that you can add to pass on to build steps. It is one of the few places that you need actual build bot knowledge, and there's a section in the build bot documentation that talks about scheduler properties. Most people will never touch these. I actually need to touch these, because we have a build target called build appliance, and build appliance needs to sometimes run with the build appliance source rev set to autorev and sometimes set to not autorev. I need to actually switch this from time to time. The old way of doing it was going into the repository and changing the source rev hash. The easy way of doing it is this. One of the cool things that I'm kind of understating the power of this, because one of the cool things, and this is probably a poorly named target here, is that you can actually sit there and start hacking that force build page to force it to do things like, well, email this alert when complete, or email me when this is complete, so type in an email address, and I have a special build step in the background. I want to build that in PC. I want to build this with CoreImage LSB. So you can start doing really cool things with this. And you do it all through a config file. As a matter of fact, I'm going to show you that config file right now. This section right here is all that's needed for that. Now, for it to do anything on the back end, you do need to create a custom build step. And then the last thing is the thing that does the heavy lifting. This is the last property in steps. The OctoAutoBuilder provides very basic general build steps that we utilize. These are generally taking build bot build steps and sub-classing them out and overriding certain methods. So to get a better idea on this, and we'll show you this in a second, you really want to read what the init is on each of these build steps. We're definitely going to go through this. Args that are used by your custom build step are passed in as our dict. Args that are passed on to whatever that sub-class is are passed in as keyword arguments. We do this to keep things separate. We're going to take a look at this now. This is the Hello World build step. It doesn't do anything other than echo Hello World with whatever you pass it into it. In theory, what you could also do is have a custom property on that build page. It says enter your first name, enter your last name, and have it passed into this custom build step. One of the reasons why if you look... Let me actually get into this because I believe I have it in my slide. Oh, there, look. One of the reasons that we have this is that this is the one thing we require. Everyone, when you override it, override it with that as a start. Pass itself, pass the build factory, pass your arg dictionary, and if you need to, pass a keyword argument. The reason we do that is because custom build steps are dynamically loaded. You write your custom build step, you drop it into your build step directory, you reference it within your configuration file. It gets loaded automatically. You don't need to hack any actual auto-builder code itself. Unless you do something weird or stupid. And one of the examples of this is checkout layers, which I wrote. You get to determine whether I did something weird or stupid. Checkout layers is a single step. But what it does is it creates multiple build steps. So for each repo within repos, it'll check out that layer. So, like I showed earlier, there is this mix and match repo concept that I needed to implement. Trigger builds needed to have the same branch and tag in the repo as the build set that triggered it. We actually, the auto project sees this a lot. So we have people who maintain different layers. If I want to build 1.4M4 branch, I need to contact all those people who maintain those layers and say, hey, yes, create a 1.4M4 branch for me. This would start as more and more people are maintaining different layers and I'm building more and more of those layers. This basically put me in the position where I dictate other people's branching policy. Which I wanted to get out of that business and I wanted to get out of that business and in general, it's just kind of icky. Some more features that got added are, and I showed you this early, triggered build steps will actually inherit the property and repo of what it triggers and render that on that force build page. There's also some more config options within the autobilder.comp file, specifically BB number threads and parallel make. Before that was just hard coded. Now you can actually override what that is. And like you saw in this, there are some UI changes. If you've seen the autobilder and if you've seen Bearerbone's build bot, this little bit of very ugly jQuery actually meets it so that you don't have to scroll through five pages worth of properties. So this is one of the things that was requested during the first showing of this publicly. I showed one of my co-workers this and he's sitting right there and he's like, I don't want to see five pages of properties. Some of the other features that added, we've upgraded build bot. The autobilder was initially on build bot 0.8.4, patch level one. We've also upgraded a few dependencies. So when you check out the entire autobilder.get repo, the only thing that you need on your actual host machine is Python. All the other libraries come down with it. So now I have a system that produces a quarter terabyte worth of data, not including all the other stuff it produces. It runs on a cluster of ten high-class high-end server machines. And Dave's not going to give me $100,000 to replicate all that infrastructure. So does it... Does anyone feel my pain here? How do I test this? Remember that code-compiled test? Well, my code-compiled test loop is six and a half hours. If I'm doing it on a single machine, let's assume I have all that space, I'm looking at about 42 hours' worth of code-compiled test loop. So this is kind of the answer. We're going to be doing this in production. It's not as scary as it seems. So next week, what we're going to be doing is we're going to be doing side-by-side installs. We'll be using different ports. So we'll have two YoctoAutoBuilders running. We're going to do the new root factor builds late night when no one else is awake, hopefully. Or at least no one else is awake with utilization of the YoctoAutoBuilder. And then what we're going to do, what I'll be doing during the day is verifying... No, wait, no, I... Don't give me creep. I'm not going to be sleeving is the answer. Yeah. So we're going to be doing side-by-side installs. This may actually slow down some of the stuff that's going on the main auto-builder, but we're going to keep everything jailed out. And then once we get through our milestone five period and everything's co-facetic, hopefully, and everything agrees what the main auto-builder has produced agrees with what the new software produces, we're going to switch over to the new stuff. Now, and this may make my manager cringe. I want this out for 1.4. I want this to produce the 1.4 of release. I'm not going to push it so that it jeopardizes the 1.4 release. 1.4 of release is way more important than auto-builder code. So like every half-faked, not-quite-finish-code project, there's a to-do list. And I am more than happy, hopefully, for people who, if they want to start working on this with me, especially if you're a release engineer and that's one of your passions, to start helping with this. There's a few bits of re-implementation details that need to go through. Persist B, generally, we maintain that between builds. ADT installer, we generate two ADT installers, a QA ADT installer and the regular ADT installer. And build history tracking, we need to reimplement as well. We're still missing a few build sets, mainly some meta-intel machines are missing, OE cores missing. We have documentation. Believe it or not, I actually sat down and wrote documentation. I know, that's amazing. A software developer who wrote documentation, it's crazy. But I did write documentation. It could definitely use some help. It could definitely use, you know, maybe about twice the size it is and a little bit more in detail. And I also need people to test like Matt. I said, I can't tell whether it works just on my configuration. I can tell whether it works on my configuration. I can't tell whether it doesn't work on yours. I have no idea. There's some more features that need to be added. One of those is killing build triggers. There's this annoying little bug that's been there for, I guess, the past year and a half and I haven't had time to fix it. That if you trigger a build, it spits out all the other builds and says start building them, and then you want to kill stuff. Well, you can kill all those triggered builds, but that one build just sits there and waits. And it waits. And it will not allow you to do anything until it either times out or you restart the server. And by server, not the actual hardware, but the build bot server. I ensure this is about ten lines of code. I just haven't had time to look at it. There's also a really cool new feature and if you follow the Yachto list, ptest integration. I talked about this in Yachto Developer Day. There's a lot of packages where you do make, configure, make, install, make tests. There's been some work to strip out those make tests and be able to, and there's still some work that needs to occur here, but be able to run them in QEMU on the actual auto builders. I know there's been a lot of people talking about board farms. One of the things that I would like to see happen is, especially if you do have a board farm, start working on custom build steps to take the end results of these builds and spit them out to your board farm and maybe spit back tests. And also if you have other use cases, like I said, I'm trained to make this so that it's very user driven. The unfortunate thing is that there aren't very many embedded release engineers. So this should be something that, if you're running a small shop, start using it. If it's not meeting your news case, let me know. And there is some upstream work that I need to do for one single patch about this build order thing, which is I'm not the only one who's complained about it. So live demo and I've kind of actually done live demo throughout this, but I'm going to show this a little bit more. So let's say, hey, Sol, do you have a mutt ready? What is it, STW mutt? Hmm? Age master under test? Is that a valid branch? Yeah, I don't remember. All right, you check for me. So I want to build master under test, which when we say master under test, one of our processes is when a patch comes into the auto mailing list, Sol over here grabs that patch and starts trying to test whether that patch is going to blow master all the hell. Are they underscores? If the patch is good, and he'll run master under test, we call it mutt, if the patch is good and passes the auto builder, then he sends a consolidated pull request to Richard, and Richard will end up pulling that up into master. So the wireless here is probably not going to be very happy with me, but let's hope. So now we have nightly master under test, but we have all these other build sets here that don't know about. Now we have to set all the triggered builds to the same repo branch combo. We just did that by hitting that button up top. So they all know that we're going to run a pocky contrived stage master under test. Let's say nightly non-GPL. I actually don't want to do that. I want to run pocky, and I want to run it with master. And I want to do that with lowercase. And, oh, build appliance. Build appliance. I don't want it to auto rev. I actually want it to default to what's currently in the recipe. So what I'm doing is I'm passing this property that is really in the build appliance config file, but I'm inheriting it all the way up from nightly. And then I force build on this, and hopefully nothing falls down. So this will take a minute to update. What it will end up doing is it will update all the repos because it nightly needs to do its own checkout. And then it will run a preamble, create an auto comp, create a BB layers comp. And then what it does is it hits this target called build images. The reason I do this is because I have a shared D-elder. Nightly goes and just does a universe fetch. That way all the other builders don't actually need network. So what this ends up doing is it ends up saving some time because the builders just go and they look in D-elder and they say, okay, I got the most recent. I don't have to touch that. And then once it does that, it will start triggering all these builders. One important thing here that I want to show is build appliance has a property. It's called wait for finish. Build appliance can take a long time. I don't need to wait for build appliance to finish in order to finish up nightly. So I've set wait for finish to be false. And this ends up going into the create images and says, oh, no, we don't need to do that. Or build images. And this says, yeah, we don't want to trigger this or we don't want to wait for this. So that's what I have. I'm sorry if Ross wasn't here. But any questions?