 Hello. Good afternoon. Welcome to my talk. Thank you for coming. I hope you had a good lunch. I'm really happy to be there talking at ELCE for once in my home country, friends. It's nice. And as I do most of the time, I'll be talking about build routes and this time it's going to be what's new. Before we get started, maybe a quick poll. Who is using build routes today? Well, cool. And who is using open embedded or some of its derivatives? Another fraction of the room. Open WRT, maybe. Whoa. Okay. One or two. Okay. Cool. Thanks. So my name is Thomas. I work at bootlin. We do embedded engineering and trainings. We have all our training materials available for free under a Creative Commons license. And we do numerous boot order, canal build system, engineering projects for our customers. And even though ELCE is in Lyon, I come from Toulouse, France. Hopefully one day it's going to be in my city. But it's close enough. And I happen to be one of the commentators of build route. And I guess there's at least one of the other commentators here. And I don't know if our note is in the room. Maybe not. But it's around at the conference. So I saw that like two-thirds of the room is already using build routes. I'm going to go fairly quickly about that. What is build route? It's an embedded Linux build system. So the point of this tool is to automate the process of building a complete Linux system from source using cross compilation. So the obvious target is to build embedded Linux systems for embedded targets. So we have an Intel build machine on our desktop. And we want to cross compile all the pieces of software that we need on our target, starting from a boot loader, a Linux kernel image, and a complete user space, which can be more or less complicated, ranging from a very simple buzzy box space, user space, all the way to a more complex user space with Wayland and GStreamer and QD and whatnot. So with all of that, in an automated fashion, you simply define what your system looks like, should look like, and it will automatically fetch, configure, build, install, and aggregate all those software components together. So it's a fast build system. You can build a fairly simple root file system in a minute, which is a very nice thing. It's easy to understand. It uses K config for configuration, just like the Linux kernel. And it's written in Mac, which is not necessarily easy to grasp, but at least is a well-known technology that most of the folks in the Unix Linux ecosystem know about. So that's nice. It's not a special language or a special technology you have to learn. It's something you probably already know. It produces small root file systems. The default file system we build, which is a buzzy box-based use-alipsy file system, is just two megs. So instead of having something big that you have to get down to reach your target size, it's kind of the opposite. You have something small at the beginning and just add up what you exactly need. We today have about 2,500 packages, and I think we're approaching 2,600 packages for various pieces of user space software. And as I said, it can be ranging from really simple libraries all the way up to fairly big software stacks. And this is basically the main thing that we are maintaining, this corpus of packages that we try to keep updated, and that is growing over time thanks to contributions. This build system produces file system images, so it outputs like an EXT4 or SquashFS or UBIFS image that you can flash on your device. It does not produce a distribution in the sense that it does not have the concept of binary packages that you can upgrade, install, remove individually. It's just one fixed system. If you want to update it, you update the whole system. It's a vendor-neutral project. There's no single commercial company behind it. It's really a bunch of open-source people working for different companies, either services companies or product makers or hardware makers who get together and create and improve that build system. The community is very active. We're going to see that in the next slides. We have stable releases every three months. And it was started in 2001, and I think it makes it the oldest in maintain build system. I'm not exactly sure, of course, but it's probably among the ones that exist today, the one that has been around for the longest. So that's pretty much what build is in the two-minute summary. So today, what I want to cover is the activity in the project for the best more or less two years. So covering between 2017-11 to 2019-11, which is not yet there, but we're approaching that. And talking about things like community activity, the release schedule, architecture support, toolchain support, we've improved in the package infrastructures, in the download infrastructure, some interesting package updates and additions, talk about reproducible builds, top-level parallel builds, some tooling improvements we've done. So I'm going to cover all of these topics in that, what's new talk. So the community activity, I very often like to look at that just to see how the community is going. Is it increasing, decreasing in size? Is the activity increasing or not? So I have a few graphs here that show the number of commits per release, all the way back from 2009, which is when we started doing stable releases. Before that, it was just like rolling release since more than two years, 10 years. So we have do stable releases. You can see after a ramp up, we kind of stabilized around 1200,000 commits per release. So it's not decreasing or increasing. It's mainly a stable project from a number of commits.q. And we'll see that it's in fact the same for the other criterias that I'll be looking at. Number of contributors per release, we have about 100 different contributors per release. And as you can see, it's fairly stable. Sometimes goes up to 120, sometimes a little bit below 100, but it's fairly stable over the past five, five years approximately. So it's a fairly decently large community, not of course as large as the Linux kernel, but the scope of the project is also much smaller. But it's in, I guess in the range of many of those mid-sized open source projects. Emails on the mailing list, we have approximately, let's say 2000 to 3000 emails per month. So it's a fairly active mailing list. So we have a development process similar to the one of the Linux kernel in the sense that patches are posted to the mailing list, review upends on the mailing list. And we have patchwork tracking our patches. So this also explained why there is so much mailing list activated, because there are lots of patches, lots of discussion going on, and all that adds up to the number of emails exchanged on the mailing list. But it's, again, fairly stable. We can see some spikes and then some don'ts. Usually summertime and Christmas time, we have a few downs. And when it's release time, we have a few ups, which I guess is matches with the release schedule of the project. So a fairly, I would say, active projects and in a stable state of development. In terms of release schedule, we've had for many, many years, basically all the way back to 2009-02, which was our first stable release. Four releases a year in February, May, August and November. And we've been doing that for 10 years with no exception. We have, therefore, three months development cycles. And we do that a little bit differently than the canal, because, well, the project is different, of course. We have a two months development period where we can merge pretty much any change, a big or small. And then after that, we have RC1, which kind of says that's the end of the development period. And we have a one month stabilization period, during which a few RCs are released, and then we get the final release. So same idea than the Linux canal in the sense that RC1 ends the development period. But the ratio in terms of duration between development and stabilization is a bit different. So it's been going on like this for about 10 years, as I said. What we've added in the recent years is the concept of LTS, so long term support release. Until then, once a release was done, it was kind of a give and forget situation where it was released. But if you want new updates, especially for security fixes and bug fixes, you would have to simply keep up and update to the next release. So what we're doing now is that every year we're picking up the O2 release and maintain it during one year for security updates and bug fixes. So one year is not yet very long term, but it's already an improvement over three months. And it's proving to be already quite some work to do that. So this allows companies who want to build product space and build it to know that every year they have this update schedule that they can rely on and base their organization and planning on. So in all of those O2 branches, we have, as I said, maintenance branches, and we push commits and make regular point releases more or less every month. So I've got some statistics here where you can see that the 2017 O2 branch adds 11 point release, about 100 commits, and same for 2018 O2, which adds 12 point releases, about 1,000 commits. And so, of course, the 2019 O2 branch is still being maintained today until 2020 O2, at which point we'll move to that one. So this is a new thing that we have. It's been working very well so far. We're seeing a number of people and companies using those LTS branches, reporting bugs, issues, or helping simply maintain those branches by submitting patches to fix security problems that affect those slightly older versions than what we have in the master branch. Architecture supports. Of course, we've followed the train of RISC-V appearing, so we have support for RISC-V 32 bits and 64 bits, supporting Jellipsy on both of those bitness, and I think Muscle on the 64 bit variant. The NDS 32 architecture was also added. It's an architecture designed by a company in China. So both of these architecture have been added and people are working on fixing build issues that fall out from the addition of these architectures. We've added support for new variants of existing architecture like the new ARM core, the new X86 core, simply flowing the new CPUs that come up from existing vendors. We've dropped the Blackfin architecture. It was dropped from the Linux channel and was anyway difficult to maintain and barely used, so that is something that we got rid of. That means that today we have support for this complete list of architecture that you see here. Of course, the big names, ARM, X86, ARM64, and so on, but we also support more niche CPU architectures. CSKY, OpenRisk, NIOS 2, MicroBlaze, all of those CPU architectures that are very often not supported by other build systems. I think OpenEmbedded doesn't have support for as many CPU architectures. We do have support for. On the tool chain support side, so BuildWord can build its own cross-compilation tool chain by building GCC and VINITALs and the C-Library. There hasn't been much change here in the features that we provide, except that we keep things up to date with GCC updates and VINITALs updates and new C-Lib C updates and muscle updates and so on. So everything is pretty much up to the latest stable version or close to that. So that's something that is kept regularly updated. And there is some very useful testing work done by Roman Naur from Smile using the tool chain builder project where thanks to CI we are able to test when there's a new GCC release on almost all the architecture we support, at least the ones that have QMU support. Use the new GCC release, build a complete Linux kernel plus user space system, run that under QMU, make a few checks and that on all CPU architectures with all the C-Libraries that we support, all automated in CI. So that's very good effort that has led to a number of contributions in our bug report in GCC, VINITALs or C-Libraries pointing A, this new GCC release breaks muscle on this weird CPU architecture. So all of that is, of course, very useful and helps improve the quality of Pyotrout. The other thing that Pyotrout can do with tool chain is reuse existing tool chains. You have one provided by Ovendor or simply you don't want to spend the time building a tool chain. You can use pre-built tool chains. So we've added the tool chain from ARM because they didn't exist back then. So they have tool chain for ARM32 and ARM64 was little in the beginning. So all of that was added. Just go in menu config. You can use those tool chains. We've added support for ARM64 beginning tool chains from the Naro as well. The NDS32 tool chain was added as part of adding these new CPU architectures. And generally they've been updates to the existing external tool chains from the various vendors that we support. Another thing that has been added, fairly advanced feature, is the capability of declaring external tool chains from BR2 external trees. I'm going to give a little bit of background on that. BR2 external is a mechanism that was added a couple of years ago in build-root which allows you to declare build-root packages, dev configs and a bunch of other build-root artifacts outside of the main build-root source tree. So this allows you to keep your own custom packages really cleanly separated from the core of build-root itself, which some people find it easier when you have to update build-root as your packages are nicely separated. And external tool chains could not be declared in such external trees because of the way it works internally and it's something we have to improve and now you can declare your own custom external tool chains in your BR2 external tree, which again helps you cleanly isolate your secret source from the main build-root source tree. Package infrastructures. So in build-root we have the concept of package infrastructure which basically factorized the common logic of configuring building and installing packages. When you have 500 packages that use the other tools you don't want to repeat the source of describing dot slash configure blah blah blah blah for 500 packages you factorize that in a common place. So we've got infrastructure for most of the major build systems like other tools and CMake and things like this. And we've added two in the last in the past two years for go-long packages and probably more importantly the meson-based packages as well because this is becoming a very popular build system so many software components are moving to meson so we support that as well. So it looks like this so this is dockercli.mk so this is the entire make file and build-root that allows to build the docker command line interface and it's written in go. So we declare a bunch of variables describing which version we want to use from which GitHub repository we clone it which license it uses what are the dependencies and then a bunch of go specific variables that we define and that are used by the magic macro at the end goal line package to expand to the set of steps that are needed to configure build and install this package. Similar example but it's time for the meson package infrastructure and actually a simpler example here for LibMPD client and the principle is very much the same you declare where to grab the source code from so in that case it's just a terrible over HTTP what is the license of that package and that you want to install it to staging because that's a library and then simply invoke this magic macro meson package that again expands to how to configure how to build and how to install that package and all that knowledge is factorized in the definition of the meson package infrastructure so it's not duplicated over dozens if not hundreds of packages so that's these are the two package infrastructure that we've added in recent times. Another thing that has been improved in the core infrastructure is the download infrastructure so as seen below there's been a major rewrite of the internals but it's not so useful for the end user the main improvement from an user point of view is get caching so I need to again here give a bit of background so before that's when you were specifying a git clone so this one is a bit special for my github it's not really a git clone but a package that you need to fetch from git like your linux kernel tree for example you say it's available from that git repository and I want to fetch tag this this this what build would do I would clone that repository check out the version that you specify create a tarble out of that specific version of the kernel that you specified in your configuration throw away the git repository and then use the tarble to do your builds so as long as you build again and again the same kernel version we're good because we have the tarble locally cached and we don't go back to the git repository but the next time you change the version of the kernel that you want to build in your in your system and during development it happens usually quite often what build would do is oh I don't have this kernel version locally available as a tarble so let's go to the git repo do a full clone check out the version and so on and you can imagine doing a full clone of the kernel each time you change the version for just grabbing two or three more commits it's not very efficient so we've changed that and we know do I think what is the obvious thing that we should do keep a local cache of the git repository so that the next time you try to fetch the kernel source code or of course it's valid for every a git based project we have a local clone available and we just grab the additional objects that are missing use that to produce the tarble and we move on so we've moved from a very long repetitive complete clones of repositories to using a local repository and only grab the missing git objects to achieve well the specific version of the software component you wanted to to clone so the way it's organized is shown here we have is the mouse working yes so we have a deal here which is the top directory where all the downloads are stored it's not organized in sub directories so for those of you who are familiar with bitwood it used to be like this big dump of tarbles and stuff so now we have one subdirectory per package which makes it a little bit nicer so this is an example of a package that always fetched through like HTTP or FTP and we have tarbles so just the tarbles are nicely all grouped together in a folder but you boot is sometimes fetched as complete tarbles as you can see here these are the official versions but sometimes in this example we sometimes fetch over a git so we have a git subfolder which contains the complete repository of that project including the dot git metadata and we use that every time bitwood needs to grab a given you boot version and produce corresponding tarbles so here ashes were used to identify the versions so that's slightly organization of the download directory we were careful to keep backward compatibility with previous download directories all that kind of stuff is handled but the organization is a little bit different now in terms of package updates and additions this is actually where most of the activity in bitwood occurs maintaining this set of 2,500 packages is a lot of work and requires contributions for a large number of people but it's it's mostly pretty boring right it's just new packages being added to support another Python module or updating packages here and there so I can have tried to extract a few highlights of that so we've added 378 packages remove 56 packages but 30 of which are x.org proto packages so xorg has split packages for all the header finds and now they have a single one replacing this so it's kind of screws up the number of 56 packages being removed we removed Qt4 it was long kind of deprecated and we had to move on and keep on the Qt5 among the things that we've added that are kind of it's a major we added support for REST so compiler itself and package manager as well we added support for LVM clang so not yet as a compiler there is work being done around that but for now it's only to have clang on the targets for things like OpenCL but not yet to use as a compiler for building the entire system we've added the OTA software Mender, OpenGDK to have another Java VM on the targets the OpenRC init system the Opti Secure TrustZone side OS and zillions of Perl and Python modules and when I say zillions it's like really zillions of more additional Python modules and Perl modules most of the major software stacks have been updated so we have Qt5.12 latest x.org latest gstreamer Wayland Weston Kodi and there's probably more that I missed because reviewing all those comets was kind of a boring thing to do and extract the highlights is not necessarily necessary easy so I just had a quick look at our history of comets and whenever a package is updated we usually do package foo bump to version XYZ in the commit title so I look at all the comets that add bump in their title as a kind of an approximation of how many comets were done to simply bump packages and it's about 3,000 comets that were done in the course of a little bit less than two years to keep our set of packages updated so most of the work is actually going on keeping packages up to date and adding new packages let me check the time alright another thing that we've added are hardening options so support for increasing the security of the binaries that we produce against certain classes of vulnerabilities so stack protection railroad and buffer overflow detection with the 45 source options we all of that was added so you can build the firmware that is more secure binaries so it's quite nice and we've recently added more testing effort around those and possibly we'll move to defaulting to some of those options be the default rather than no protection at all but that's like on the rider for the future another thing we've added is make show info it's a target that spits out a JSON blurb that looks like what you can see on the right side of the slide here it basically tells you details about all the packages that are enabled in your current configuration the name and the version and the license and all that kind of things and then you can do whatever you want with it parsing it to write a nice HTML page that describes what is in your products or download the tarbles that are needed to reproduce the build of that product basically whatever you want there's just some metadata about your configuration that you can easily parse because it's JSON we've done some work around reproducible builds where the idea is that given a build-root configuration version if you do two builds of the same configuration you get the same binary identical result so there was a Google summer of code this summer with Atarva Lele I'm not sure I'm pronouncing correctly working on this topic and he was mentored by two core build-root developers Arnaut and Jan and on improving the support for that in build-root there was already some initial work that had been done in the past but it kind of pushed that a bit further and it's mainly on the testing side it has been pushed a bit further because our build infrastructure that already existed has been extended to test reproducible builds so what we do is that for some builds we do the build two times and we do it in two different folders so that we test the different we test builds that have a difference in terms of time and build location so for now the build environment is the same it's the same build machine that is used to do build number one, build number two but time and build location is different doing builds in different environments is like something we would like to do but in the future we kind of take it step by step and not introduce too much randomization at first and first fix those issues number of issues in tar, jizip, CPU handling around timestamps were fixed around paths as well but of course more work is needed but we do have reports like that in the automated fashion so when a build fails because of reproducibility issue so basically the build succeeded but the file system images are not binary identical so we compare them with this tool called Defoscope which investigates the file system image in that case it's a tarble it sees the difference in that case in a shared object from the asterisk software and then uses really ELF to go inside the binary identify very specifically which part of the binary has a difference and you can see here output one, output two these are the separate folders we use for build number one, build number two so it sneaks in into one of the binaries someone needs to figure out why and how to fix it but at least we detect it and we have a very clear idea of what the problem is another topic that has made some amount of progress even if not enough is top-level parallel build so right now the way it works is that builds are linear every package is built after the other within each package we leverage multiple cores by using make minus j something depending on your number of CPUs but packages are built one after the other and what top-level parallel builds allows to do is to do this use your multiple cores to build multiple packages in parallel of course as long as they don't have any dependency relationship so this is something that has been in the works for quite some time and the main thing that remains to be done is and that for which we already have patches so it's really in the works is per package directories where the idea is to isolate the build of every package so that even if other packages get built in parallel they won't interfere with the build of other packages so without going into the details it's really isolating the build of each package in its own little environment so that having parallel build is not going to close confusion in the build of other packages so in this example that build which is not very representative it doesn't have that many packages was about more than 500 seconds and with top-level parallel build enabled it was down to 300, I don't know, 70-ish or 80-ish seconds so it's already a good improvement I think the improvement can be better with bigger configurations which have more packages that don't have dependencies with each other because we can see here there's a chain of dependency here where no parallelization occurs because this depend on that which depend on this which depend on that which depend on this so this is something we're working on it's not yet there but hopefully 2020 will be the year of top-level parallel build it's just like it should be the year of Flonix on the desktop runtime test this is another thing we've added in 2017-02 so it was already there and compared to the two-year time-spine I'm looking at but it was improved quite significantly so this runtime test infrastructure basically allows us to have a set of well-defined build-root configurations that build a system for ARM with this piece of user-space software builds it, boots it under QMU and runs a bunch of commands to verify that whatever we want is working let's say this Python module or DropBear is running or that kind of things and these tests are executed in our CI so that's for now it's once a week we run the entire set of test cases and make sure that they continue to run and this runtime test infrastructure was mainly improved with additional test cases so we moved from I think a few dozens of test cases to a few hundred of test cases or something like that testing Python modules per module UM module OpenGDK and Docker and Harding Flags and lots of other features and we of course need many more but it's been a lot of improvement and if you want an easy way to contribute to build-root that's a very good starting point adding one or two more additional tests there's been also tooling improvements so next there was a Google summer of code on one side with Atarva and on the other side there was an internship in France at Butlin where Victor Oeskar, a student from France worked with me during the summer to improve some of the tooling we have around build-root to maintain it and this is mainly around release monitoring for tracking upstream releases notification to developers and auto-pillar search capabilities I'm gonna give a few more details so releasemonitoring.org is a website from the Federa community that tracks upstream releases open source projects so it looks like this for the BuzzyBox project so they know BuzzyBox is hosted there and they regularly pull their like HTTP sites to know when new releases are made and they provide an HTTP API so we can query to know what are the latest version of BuzzyBox available so it tracks 27,000 projects and build-root has of course a lot less but even with 2,500 packages that we maintain it's difficult to keep all of them up to date so what we have done is that we have extended our tool that produces this table for every build-root package it knows the current version which is this column so that's the version we currently have in build-root and another column shows what releasemonitoring knows about that package so for example this is not up to date that's an example so this kind of gives us oh maybe we should have a look what are the improvement in that package are there security fixes there is there anything interesting so we've done a number of improvements to that script to basically query release monitoring improve the speeds of the script and other things like that so that was an initial tooling improvement that was done and this page is public you can go to autopillbrew.org slash that and you've got this page updated on a daily basis based on that we've done improvements to the notifications sent to developers so very much like the Linux channel as a maintainers file we have a developers file that says this package, this dev config and this test is maintained by this or that person with this email address and we use that to send emails so we were already using that to send email about build failures occurring in our autobilder infrastructure and we've extended that to send notifications about packages that are not up to date hey you are maintaining DropBear but DropBear is that version of this in BuildVoot and there is this newer version available upstream maybe you should have a look and submit a patch and provide an update to that we've also extended it to notified developers when the dev configs they maintain no longer build in our CI or when one of the tests they maintain no longer build or runs in our CI so it gives something like this for the version notification so it says a package ACPICA is that version this, this, this in BuildVoot and upstream as that maybe you should have a look at that so this is sent both to individual developers about the packages they care about and also to the mailing list about all packages so that the entire community can participate to the effort and here is the same idea for what I say dev config failures and runtime test failures when a dev config like for a particular platform no longer builds or when a given test and no longer builds or runs we've improved the search capabilities in our autobilder so our autobilder is a form of build machines that 24-7 generate random BuildVoot configurations to build and provide the results and it allows us to detect dependency problems version compatibility issues the fact that this given package on this given architecture with these optimization flags and these hardening flags doesn't build and we've been running that for many years and the improvement we've done is basically be able to search in the history of build results by config symbol so that we can answer questions like what were the build results that were successful on ARM with Usylipsi and that had the BuzzyBox package enabled so of course for BuzzyBox it usually builds fine so that's not a very good example but sometimes it helps us investigate since when something started to fail until when it was still working and when it started it failing so that's a very nice improvement for mainly the core BuildVoot developers when they are investigating build issues straight on the scope is it only appearing on one CPU architecture or multiple CPU architecture is it appearing only when we have this combination of that package and that package enabled or not so that's kind of helping us answer those questions so it looks like this is a very simple page that you can give all your search criterias and it gives you a list of build results we've got a few other smaller improvements that were not worse adding their entire slides make Linux diff config target that shows you the difference between the current configuration you're using for the Linux channel because you've changed it you've edited it and the one you have stored for your platform configuration so it shows you okay you have enabled that driver, that other driver maybe you need to update your configuration according to it we have support for generating images file system images in the F2FS and ButterFS file system formats so in addition to the usual as I said EXT4 and squashFS and UBIFS and finally we've added support for get text tiny as an alternative for the full blown GNU get text so if you don't need complete native language support with translations and you can work with just a stub that does no translation get text tiny is nice it has a smaller footprint, smaller build time so we no support bus of these alternatives for the get text implementation so to conclude project is active we've seen with the comets and a number of contributors we've got LTS release once a year with security and bug fixes a few new CPU architectures new packet infrastructure to cover new build system that show up the get caching feature in the download infrastructure the fact that we keep packages up to date we have many new packages and new notifications around release monitoring is even pushing that forwarder and we see a good increase in a number of contributions in this area because people receive notifications so they submit patches to help keep their packages updated reproducible build effort in progress they've been a good push and we hope to continue that effort and as I said, maintenance tooling improvements as well we're done there is a tutorial on build routes on Wednesday if you're interested about the basics and other than that, I'm about done I don't know if I have time for questions no, maybe I'll take one just to pretend I took questions and otherwise, I'll be in the hallway for the entire conference so feel free to hang at me and ask questions maybe one question and then I give the microphone to the next speaker thank you for your attention one question, yes, please so what's the best way to do an NETROM FS with build routes? I was hoping for a quicker question so it depends, if you boot your entire system to an NETROM FS and stay there and never switch to a root file system that's kind of easy but if you have this combination of you need an NETROM FS and a root file system and you switch between the two during the boot process then essentially you need two build route configurations one to build the complete root file system and one to build the NETROM FS and then some kind of glue between the two to combine them that's the kind of 30 second answer I can give all right, thanks a lot again I'd be in the hallway if you have more questions and I want to leave the place for the next speaker thank you and have a good conference