 So hello everybody I'm Tristan I work for code think and I'm here to talk to you about I'm here to talk to you about build stream, which is a new build and integration technology useful for integrating full software stacks and for developing software within these stacks At code think we've had some previous iterations of build tooling called base rock and the YBD and this is basically an extension or a new Version that we're using to build with in GNOME and flat pack and etc. etc. So let me get started because I'm a little bit late and flabbergasted and the slides will help me get on top of my game So what is build stream? Trustable build and integration tool we because we have a focus on deterministic builds and We isolate our build sandboxes. So basically every build environment is like a production build environment Whether you're you know on a build server or on your laptop every build environment is the same Basically, it's a pipeline of file system data permutation. So the design is very simple. I don't have a thing here. So The design is very simple it's just basically a pipeline of elements related by their dependencies and each element performs Some activity on its input file system data and creates output file system data and so on it just happens to be useful for builds Sandboxed execution environment Builds don't have access to the network etc. etc. As you would expect in a production build environment We do caching and sharing of build artifacts, which is interesting for work groups and for and for CI and for a combination of both for instance a work group might use a CI server who's doing production builds or just test builds a lot and that means that since they're Populating an artifact cache When a user or a developer builds then most of what they want to build is already available And they can just download it instead of building Multi-purpose build instructions and metadata so Perhaps you've been in this position where you have an application and you want to distribute it on several targets Say you want to distribute a flat pack you want to distribute Well bad example because we don't really support windows yet, but you know you want to support a Windows bundle or You know or a Debian package and you have to have this build directory with so many different scripts of All the different ways that you build your software and We want to be able to allow you to do everything with the same build metadata So there's that and it has a developer story We'll get into that later. All right, so what are our motivations for developing build stream? So looking at this and as a perspective of you know We weren't satisfied with build root. We weren't satisfied with yachto And we we're coming from a perspective where we want to build custom Linux firmwares for embedded systems, right? so one of the things that we observe is a lot of work is duplicated in in cross compilation in different projects such as build root and yachto and pushing patches upstream to High-level or middleware, which doesn't really need to work with cross compilation So I wonder if it's even worthwhile to spend all of that human energy patching upstream stuff patching middleware to Be able to cross compile and patching their configure scripts and their build their build files Seems like a big waste when all we really need to cross compile is a base runtime and a kernel From there we can basically just native compile or virtualize and save ourselves all the whole headache But the same time build stream doesn't really prevent you to do cross compilation. This is just kind of my you know opinion One of the Motivations is smoke testing builds on new host platforms. We don't want to have to do that, right? so we just say that We're never going to touch the host platform. So Why bother smoke testing your whole build on a new version of Ubuntu or a new version of Debian or a new version of Fedora where everything just worked the same, right? It's in the later slide, but basically we have a no host tool policy and build stream You can never build on your host You build on your host, but you never use Your host compiler you never use your host libraries. Everything's always containerized, right? and Another point is complicated setups for production builds. So since build systems generally are a byproduct of a Distribution right The build system is just basically Well, we needed to also build this so These things are like You know RPM and Debian. They're more than 30 years old now And they come from a time where we had a very different set of needs. So people still get to do production environment build Build setups, but then they use OBS and you have to like set up a server and everything and you have to You know integrate three or four technologies together just to get something that is more or less repeatable and reproducible as a build environment and that was one of our motivations Monolithic repositories of metadata. So if anybody has anybody in the audience used build route or Yachto. Oh That's that's not very many people. Okay. Well Generally When you're in these systems, they are the more popular systems for generating custom Linux firmware So if you if you want to any device if you want to make a modem or not a modem But you know if you want to make a set top box or Or anything basically based on Linux You're going to use one of these tools to target your platform and to target your hardware and to build something Customized and to take Linux libc the compiler and You know all of the core rutils and everything you're going to build it in this one system right and Buildstream is kind of another one of these except you can use it for other stuff, right? And this is the build space that we're working with Typically these repositories or these systems come with a prepackage set of Instructions to build this system so they already have a knowledge of what they're going to build which is another thing we don't really like but They're also monolithic repositories, which means that you have one repository with all the build instructions to build basically everything that you could possibly build right and that means that say if you have a You have like a system or or a base image where you might want to use cute, right and cute five and Wayland right Then you're gonna have that all in the same tree with the Linux kernel and with With g-libc and then when you make a modification to g-libc or or to how you build g-libc or how you build the kernel and It's gonna have a repercussion on all the different variations of things that you can build right so this kind of creates friction in a huge environment like Yachto where you have like so many different output platforms and Yeah, so Note on the time So basically the the fact that we had all of this in the same tree was something that we didn't like so we wanted something That was more flexible which allowed upstreams to Downstreams to consume from upstreams so upstreams would be lower level lower level stacks and people would maintain those and upstreams could consume them and then if we were developing a desktop environment then we could just Decide when to depend on a new version of a of a runtime So I'm gonna skip over this because it's basically the same thing tight coupling of build systems and distributions means that you know, we have this Build system, but it's only made to build this and I can't use it to build that basically because It's ingrained and it's built together and It's not flexible So what are developers? What about the developers, you know Generally developers work on their laptop work on a on a module test it on their own and push the stuff upstream and say okay, it's fixed or if you're in like a In a company then you might have higher bar of entry and a lot more restraints But you're making the developers life difficult by saying that you have to install this device And you have to deploy to this device and we want to bring that Introduced that Integrated production environment directly to the developers workstation so that the developer can actually test within a target product and within a targeted environment and We do the best that we can within a container shell but You know, of course at a certain point you have to boot a VM or boot on hardware So I'm going to skip over this because it's not important Cross compilation So I basically spoke about this already production builds everywhere by default Let's just take the environment bring it to the developers laptop So we have a high focus on reproducible reproducibility and repeatability Repeatability of the same process is to say that we can confidently repeat exactly the same process Every time that we build something Reproducibility these days means bit-for-bit reproducibility of the output So for a given set of inputs, we know that the output will always be bit-for-bit identical That is not something that we can achieve Because if you look at the reproducible builds Project, which is actually very interesting There's a bunch of reasons why source code also still has to adapt to be able to be reproducible but We do by default provide an environment that is most conducive to creating reproducible builds When we say minimize on host dependencies for the build tool, that's important for us because we think going forward It's important to be able to always bootstrap a new hardware and get to a point where you can start building and You will have that same experience on a new platform so Reusable multi-purpose build metadata is as I said before important to be able to target multiple Outputs and multiple packaging formats with your same software, right? Right so Your developer cycles within build stream would be a little bit slower, so I say quick here, but scratch that It's a little bit slower But at the same time it means that every time you launch a shell or you run a test You're running on exactly something that you know and you're not necessarily only testing your application You're testing your application with a Modification that you might have done to GTK or to cute or a modification that you might have done even in G-Lib C and then you can just run it, right and Always know This is exactly the combination of things. I'm testing right now in this shell, which is quite interesting so artifact sharing is something that the belt developer benefits from and If you have a work group and somehow I'm getting a bit redundant here. Sorry about that artifact sharing is People are appreciating it a lot right now in free desktop SDK because we're building things that take seven hours to build and When we share the artifacts then we only need to build like you know for 30 minutes and 20 minutes of it is downloading right because we only build what we want to test and what we want to change So I have enough time I ran through that Let's have a little look at what this looks like so when you this is a very small sample project it's glade Running on an import of a Debian Basically image So we we're going to import a Debian image and Then we're going to build glade on top of that image Using the tools that Debian provided and then we're going to use Debian packaging tools to create a Debian package of glade all Within a containerized environment, which if anybody were to run on their laptop They should get exactly the same result basically, so if you were on a fedora system or an Ubuntu system or Suzy system or any system that you could run build stream you should be able to produce exactly the same Debian package for exactly the same Debian version right and One thing I'd like to just point out in the show output here is The cache keys These are very interesting in build stream. They are basically hashes of all of the input so every time We see the hashes a lot in the user interface because they represent exactly the version or the The the hash of all of the inputs for a given output basically so I'm just going to go over Okay here I don't want to spend too much time talking because they don't have that much time So basically projects are composed of elements elements are all YAML files right now Elements are related through dependencies and Elements can depend on elements in other projects that in that allows for separation between projects and Maintainerships of different builds and Elements can all be implemented by plugins So it's a very flexible system which just provides something for pipelining of builds in a sandbox environment right a Pipeline is what we call a loaded collection of elements So this is maybe one example of what a pipeline might look like if you were to visualize it import base system build a lot of stuff compose it into one File system and generate an image with it right so those are all different kind of plugins that we have we have an import element That's gonna like just take an import of some data and At the base of every pipeline there's always an import build GNOME and so here for example we have basically the same kind of GNOME ish example here where we're gonna build like epiphany or Create a flap pack Yeah, so this is a Integrate for the SDK and integrate for the platform So this is this is basically how we create the GNOME SDK, but like in a very condensed summary Take a look at how the YAML files work So here is basically what an import looks like Seems the resolution is not very big I'm not sure if anybody can read this But basically you have an import you say this is an import element by specifying the kind and Then you say this is the source right so you have element plugins and you have source plugins and Sources are just basically things which can interact with different revision control systems and tar and Anything to get data in a reliable hashable way Basically, so then we have this so since we imported Debian From a deep from like um, what is it multi multi strap right so multi strap creates this thing? that you can create a Debian image of Like multiple architectures right, but since we do it on On an architecture that we don't know we like that debut strap process We do it on an architecture that doesn't support running the binaries of all the different arches that we do We postpone that step until the sandbox and we just run D-package configure dash a right so this this guy just basically takes the output of the previous element and Says okay, well my install root is slash and I'm a script and I'm just gonna run this Transform my file system and the output is gonna be the slash and Basically just runs deep package configure everything Some things fail, but mostly everything comes together and it works and we're happy. Yeah Of course if you were working as we had been a year ago If you were working with this deep package configure thing you wouldn't do it that often And usually it would be in a cache server and you would have downloaded the already configured for your architecture artifacts, right? So this is basically just the build right this Yeah so here we say I depend on the configured deep package configure thing and my sources are Glade dot git from gnome Track the master branch Basically normally you would have the ref in there and that would be the git shot of exactly which shot that you're building but for brevity I kind of like snipped some things out and Here we have some Public data which says the public data of an element can be read by its reverse dependencies So this is just saying like okay. I'm giving you some data and then the later element can read it and the last element here is the Deep package deploy and we've done this also with rpm in other projects I don't have a Demo right now a colleague of mine is having a demo is creating a demo of building mudzilla Firefox browser on To for a snap and for a flat pack. I don't right now have a project which does multiple outputs This one this one basically just says my input is glade and This deep package deploy element that's a plug-in that exists in BST external repository It just reads the public data of what it's packaging, right? So there's not much configuration about how to build the package here because it's all already Specified in the package where we declared the build All right, it's a lot of material I think For a short time the status of the build we have stable releases since two years now We're not making any more stable releases Because we're working on build stream to and that's a whole other deal So we have a stable release 1.2 and we're adding bug fixes to it right now only It's used by the GNOME release team to To test and smoke test the build process of all of the All of the modules see that they build together before every release It's used to build the SDKs of flat pack. So the GNOME release team is able to Maintain the same build instructions for their GNOME release and for the SDKs for the GNOME flat pack SDK Parenthesis a flat pack SDK is like a Firmware but without any systems and without a kernel that you use within a system Called flat pack, but I mean Snap has something similar and that's basically a base that you can run things on and We use it there we use it for the base free desktop SDK as well and Right now I'm working on getting the GNOME nightly Images so you should be able to have like An image of the latest GNOME and boot the latest GNOME at any time And we should be doing nightlies of these with build stream and should be done in about a month. I guess so we're working on 2.0 and Since we've been doing a lot of work with Bloomberg and they care a lot about scale We're working on scale and optimization. So I think the project is pretty fine You can work on like a a big project. This is about 500 or 1,000 different things You know that you want to build but like they're not happy if you don't if you cannot build 50,000, you know, we want to load a pipeline and parse 50,000 YAML files and we want to be snappy, right and they're doing really interesting stuff. We're working together Okay, so also we're doing Windows and OS X ports basically I don't think that we're gonna have native builds in Windows anytime soon But we're gonna have the client working on Windows and we already have WSL tested Right distributed building so that should have been before we're working with the basal and build barn people We met up in London this year and We're working on using the same distributed build network. Does anybody know what basal is? No blaze. It's the Google build system and anyway, it's It's it's a concept to speed up builds because they have huge amounts of software to build in a short time and They have very granular units It means you have to rewrite your whole build system for your software to use basal but if you do and you have a huge monolithic repository or a lot of gets that come together aggregated into one directory It can be very interesting if you're willing to spend the effort, right? So we're working on integrating entire system builds with Google's basal To work on the same build network so that we can use the same caching resources and optimize full system huge builds With them and it's a lot of fun. I think it's a lot of fun If you haven't fell asleep yet Then you can join us on IRC on the mailing list we have about three hack fests a year and That's our last hack fest that we cannot see very well in London But as you can see you can't really see anybody who's there, but you can see that there's a lot of people there You know and and you should join us Yeah, so any questions anybody want to know about build systems So the question is is why is it important to have a build you can reproduce? Why is it important to have a build you can reproduce? Okay Bit for bit One of the main reasons bit for bit reproducibility is important is for validation because we don't know We don't if we don't know exactly what went into the output Right, and we don't then we don't know how we made the output right and once we validated something Like it can take like five years for a car company to be really happy with their base system libraries And if you say that we have to rebuild it again, but it's going to be different, but I only changed one line of code Good good luck good luck getting them to to Make that bug fix right because we trust this binary work for five years, right why change it So it's it's obviously a more important for security and In telcos or communication systems, of course and safety critical. Yeah Any further questions? How hard is it to say support for new languages in? Buildstream, I mean like I don't know no JS or other Ecosystems with lots of packages. It's very easy Basically Well, it's very easy for us in the build stream sense We really only have some yaml defaults of what to do, right? So the the more difficult part is providing a base runtime, which has the tooling that you need right, so it's just a question of setting up a project that has the tooling that you need to execute and Writing up a default yaml plugin that says well, this is the default stuff and you can override, you know Configure options with such and such things Okay, so that answers the question and so a round of applause, please