 Good morning. Welcome to the talk My name is Yussi, and I'm here to talk to you about build systems Usually when I talk to people about build systems the reply I get is this Which is that all these build systems are terrible, and why should I care? Well, let's look into this If you were in the keynote one of the points made there was to explicitly say that don't build any more Python extensions So with that in mind let's build a Python 3 extension module and To make it more interesting. We're going to be using C C++ Rust and Fortran all in the same extension module and If you've ever done this with these details I salute you But here's the height would look like if you would do the same thing with meson so First you start the thing by defining the project which in this case is called poly snake and the list of languages that you want to use and The next step is this function where we import Python 3. This is the Internal module system that we have in meson and this Python 3 module has all the knowledge to build Python extensions And then we find the dependency and this What it does it goes out to the system and finds out what are all the the flags and and linker flags and all that sort of Things that you need to compile Python 3 applications on Linux This would be going into package config and on other platforms. It does a bunch of other magic stuff Then we build a rust static library rust is a bit special. So you have to do it this way But this is very quite quite simple And once you have that then we can start the actual extension module building So we call py3 mod extension module And we give it the name and Then we list the source files that we want to have in it in this case the C file the CDP file and the Fortran file As we as you can see we're using the modern F90 and not the FF 77 and Then we tell it that we want to link the rust library that we built earlier with this And then we say that the dependencies of this project are the this is the Python dependency that is there and This is the build definition in its entirety. This is all you need to write and Then you can just compile it and then it goes and This is so We've been the design goals of this system Which were which you could see someone on the previous slide Go off it like this. So there's the the platforms that we have today, which is Linux OS 10 Windows BSD's and a bunch of other ones We want to support all of them There are several different compilers GCC client visual studio the Intel compiler. We want to support all of those We try to hide as much of the platform specific differences as possible Including compiler flags and all that sort of stuff basically just want to say I want Compiler warnings to be enabled and then we do that and we take care of all the back-end stuff for you and Out of the box, we should provide support for tools that people are using today There's very little point in having Like 7,000 line make files where you copy bits from one place to another because this worked in one project So maybe it works in this one as well, but then it doesn't and then you're just Debugging for quite a long time and The major architectural decisions of this system are not defined by what was in the default year's land of SunOS 3.5 circa 1987 because that's what like for example auto tools does it's it's an admirable backwards compatibility claim but like maybe it's time to let go and at the very core of Mezzan is the domain specific language that is used for defining these builds This is the thing that we have spent most of our energy on so I'm getting as good over definition language as possible and Let's look at some of the design calls that we have the first we're going to look at some of the design not principles because Your product is defined by what it does not do more than it's defined by what it actually does Especially if you have something that is Turing complete But the things that you specifically are saying that we are not going to do Help you make a lot better design decisions later on So the main design thing is that mezen is not a framework There are a bunch of build systems which say that well, this is not a build system This is a framework with which you can build the perfect build system for your project as if this was a good thing No, we don't do that. We have a very specifically focused. We do this one thing. We do it. Well, we do it only and The there are a bunch of built systems that are declarative languages and declarative languages are really awesome when they work but the problem is that when they don't work, it's it's really really painful to debug and Because of this with mezen is not a declarative language But we try to be as declarative as possible and It's not Turing complete You can't specify your own functions or recursion or any of that and the reason for this is that it makes tooling much More easier to write because it's easier to understand what the thing is actually doing But it's not Python It's good. It hides the fact that it's implemented in Python. So it could be re-implemented in any language at all What those were the not principles. So what are the actual design principles? Basically it boils down to this it stands every second you spend writing or debugging build definitions is a second wasted You're not being productive when you're writing build definition rules It's just boilerplate stuff that you have to do. So let's do it as little of it as possible Understandability is the most important thing You should be able to look at your build definitions like one file in isolation I have a fairly good understanding of what it does and it's not like some sort of spooky action at a distance where like It says here that it's going to do something but actually over there. There's something else which completely changes everything So we don't want to have any of that Performance is important when it needs to scale to tens of thousands of files even more and You have to have have this sort of performance built in although the core stuff needs to be properly done You can't really likes like bolt-on afterwards some sort of performance Zen of Python Explosive is better than implicit awesome All that good stuff. There should only be one way to do something and it should be obvious awesome so we stole all of that and Basically, we just steal everything that is good from anyone if there's any good There's no such thing as NIH if someone has a good thing that they're doing and we are not we're gonna steal it and we're gonna be proud about it and The one thing is that the build definitions should be shorter than writing the exact compilation commands by hand So let's look at an example a hello world Example looks like this use to define your project and then you say I want to print an executable which has this one file and this is 54 characters and if you Do it manually you said CCW all all the stuff you compile your link and that's 54 characters So we didn't quite reach the goal. However, if you use GCC, then it's 56 characters and we win We also want to be usable Across all for every single use case and the only dependency that we have is Python 3 and Only Python 3 and the standard library anything outside of that. We're not allowed to use any of that as an example, there are a bunch of build systems that are implemented in Java and and The problem there is that there are many platforms on which Java cannot run and let there's no CPU. There's not enough memory all that stuff And and it's a it's a dependency that a lot of projects can't take and and even more projects don't want to take But we want to be able to to support all of those and Python is really good because it's it's quite small It's self-contained and it's everywhere for design we've been doing the 90 slash 9 slash 1 rule which is that there's 90 percent case and that should work out of the box because almost always this is the case because you basically there's only so many ways you want to build an executable with some sources the 9% case should be possible and For the 1% case we can just say is like What you're doing is just so weird that we can't support that and it would complicate everything else completely and usually It's like okay, but if you do this other thing instead then we can just do that But it's it's totally okay for some of this like the weirdest of edge cases to such to say it okay We can't support that out of the box. We have to do some of your own magic In practice this has been less than 1% So but it's still there and There are already people using it that there they're like a hundred build systems usually most of them have about two users But we have managed to get some of them and one of them the major users Thus far has been the cheese tree or multimedia framework, which is 100 150 plugins and a base platform and all that good stuff and they're using it to build on windows and and OS 10 and Linux and all that stuff and Then there's system D For those of you who don't know system D is the first project But first process that is spawned when a modern Linux system Boots up and it's like the manager process which take care takes care of all the other processes and they actually have a Outstanding built merge proposal to remove order tools all together. So then starting from the next release They're probably gonna have only mess and for building The X or graphical user interface, so if you are Linux laptops, you're probably using one of those Using that and if you're not you're probably using Wayland Which is the the up-and-coming new graphical Google server and that has some built definitions as well And Western is the the reference implementation window manager for that The GTK widget toolkit Which is pretty big and it's multi-platform and in their four series There they they have merged the the mess and build definition and they're gonna I Think they're gonna some at some point get rid of the old one, but it's it's not not yet happening A lot of girl project are are going as well multi-less and a bunch of other ones and There's the PTV non-linear video editor, which is also implemented in Python. It's pretty cool project you totally check it out and On the website, we have a list of bunch of other projects that are also also using this and One of the main reasons why people are switching into meson is that you get Some fairly hefty compile time improvements So as an example system D on Linux went from 16 minutes to compile to two minutes and 32 seconds And if that seems weird to you, it seems weird to me I went through these lights like four times and every single time I saw this one is like that can't be right I have to re-verify that and and and every single time that it's still two minutes and 32 seconds and And and so it's it's fairly heavy heavy difference if you are compiling Wayland on Raspberry Pi it went from six minutes to a minute and There's the freedom re which is a debugging toolkit and their cross compilation time of two IOS went from six minutes to one and a half In practice what we seem to find is that projects can go up to 10 to 20 times faster if they're make-based and Especially if you are on Windows because on Windows file system operations are like abysmal is low and Because we don't use make and and we are not going to use make ever we don't have this problem and For incremental builds the difference is even bigger because there's a bunch of optimizations that we do where we can skip relinking of stuff and So we haven't done any to large-scale tests with other ninja generators like C make for example has a ninja backend and but it seems that we produce slightly more efficient files, but this is like This hasn't been conclusively proven either way there was talk about built-in support for stuff. So we support The sanitizers so address sanitizer memory sanitizers or UB sanitizers are all stuff He just likes toggle the switch say I want this enabled and then you have it coverage report pre-compiled hitters cannot build all of the things that you would expect and And and like scan build is the static analyzer that comes with clang And if you have it installed it the there's a target called scan build and it's just right run it and then it runs the thing and Reproducible builds that's another interesting thing. This is a big big thing that is going on with Debian and a bunch of other projects We basically want to prevent the generation of Binary packages which have flaws that are not in the original source So if you can produce from the same source files the same bit-for-bit identical binary files From many different people are going to do this then you can be fairly certain that the the binary that you have is actually proper and you haven't been back door and Basically want to support everything within reason Because there's no point in everyone trying to reinvent the wheel of how do I support this specific? One tool it just put do it once in the core system and then every company use it We also have a unit testing framework for running your tests which includes stuff like Setting your environment to have set environment variables and combined on arguments and things to run and all that sort of stuff and we log Standard output standard error. How long it took all of that good stuff and Excuse me It's easy to add support for stuff like valgrind and all tests are run in parallel by default Which is particularly nice if you are running valgrind because then they're about 10 to 20 X slower than they usually would be and You can run the test suite multiple times you want to You can run the test under GDP which is kind of nice as well And then you can just run test under GDP until they fail And then you just like leave your machine running if you have a flaky test and then you come back to it And oh now there's the the GDP prompt for the one the one in 1,000 runs that actually failed Which is kind of nice as well Cross compilation wheel of course do The full Canadian cross if you don't know what that means It's probably not very important But if you do know what it means then it's probably quite important for you and and we support that fully Full Linux and Linux cross compilation that's being used Lots and there are people who compile from Linux to Windows So they're using the mean GW packages that come with their distro So they can just generate all of their windows packaging from their Linux desktop machines very easily Android works iOS works and Also bare metal. So if you have an Arduino thing happening you can compile compile with meson and In this unit stuff like upload the target And these sorts of things are very easy to add and then you when I'm doing Arduino development I always run ninja upload which compiles and uploads the thing if it's everything works and There are very few changes in the build definitions that are actually isn't usually you don't need anything Yeah, and just the same build definition because all the parts that have to do with specifying that the cross environment Are not in the build files. They're actually somewhere completely separate But I probably many of you were thinking like the most obvious Question which it always comes to people's minds when we talk about cross compilation Which is what about bare metal embedded cobalt development? And I would like to say that we just we do support that so there you see There on the left you can see the cobalt program for running this External LCD screen and there's the screen running the cobalt program There and this is nice and working if you are in this that sort of thing Other other things that you could do when cross compiling is that you can specify it is sort of like an exe wrapper and This is a program that you can use your cross build files during the build time So this usually includes stuff like wine where you can run Windows programs or QMU which you can use to run like ARM programs on an x86 machine and This ties in nicely because you can have code generators that you compile and then you're using to generate more source files during the build So you can just run the executables as they are or you can specify that some of the executable should be built with the native Compiler and not the cross compiler and then you can just run them and they'll probably run a bit faster And you can also run the full test suite of your application with all this So if you're on Linux you can run your full test suite for Windows when you're running wine and As an exit somewhere well doing pretty well It's about four and a half years old And we have more than 100 contributors at this time and Which is about monthly releases And it's support on all of the major list of so so Debbie and Fedora open Sousa and all of those and they have default package libraries Sorry package repositories We run fairly extensive test suite They are the docker image that we use for testing the Linux part is about two and a half gigs of all the dependencies that It's needed to run all the actual tests and Building stuff is relatively simple But dependencies are what makes things everything hard so we spend a lot of time thinking about this and So the way that works is that in a mess and you can build any project as a sub project of a different one and it's run in a sandbox environment so it can't access stuff from the outside and It's isolated, but then from the master project you can reach into it and take stuff from there So this makes it possible to have your dependencies either come from the system as like on a Linux machine We just use the package config files to get your dependencies or you can build them yourself and The build definition for file for both of these looks exactly the same You don't really care where your dependencies come from you just say I want this thing to be available and This works cross platform. There's nothing specific about it So Windows OS 10 Android all you need to do is if you're The dependency that you're using compiles with meson on that platform then you can use it This is very similar to to rusts cargo or D has a similar thing called up and I think go has something called go get But the thing is that this definition is not tied to any specific language. It's it's completely language agnostic So you could use it for any any language dependency that you care This as a text this is not probably Okay, sorry, and we have the this called rap DB which hosts these dependencies online and This works in total cooperation with distro package So if you are a distro package here one of the things you hate more than anything else is that people embed the sources of their dependencies inside their packages And then they build those and then you have to rip them out and you and maybe they have changed it and all that stuff So this works in complete cooperation because you can just say that all I want is You're not only allowed to use system dependencies And if you're not having at those then it's a hard era. Don't try to compile your embedded source and never download anything from the internet, so if you have a Thing where you must have must build all of your stuff without accessing the internet we can enforce that This is a bit try perhaps as text so let's look at a dependency demo So this is a simple program that uses Lua and it opens a bunch of PNG files So it depends on Lua leave PNG and then leave PNG depends on Zedlib and this is what it looks like so there's the the Build definition file and there's a single C source file and then the sub projects folder which we'll look into a bit later So if we cat the build file it looks like this So basically just define the project and then you say that you have these dependencies Which are Lua and leave PNG and the fallback says that that Where the fallback dependency is so it's in a sub project called Lua or leave PNG And then you just say I want to build an executable and then these are dependencies that I have So and then the sub project folder looks like this and it all it has is these three files Which are called wrap files And if we get one of those then it looks like this And basically it's just an URL file for where you download the source files and The the hash code for verifying that they're actually the real thing and they haven't been a mitm and So there's the source file and then there's the the patch file Which in this case adds the mess and build definitions because the original project doesn't have them and Then you start run the configuring step. That's what it looks like. They see downloads with PNG and it downloads the patch and it does all the patching and As you can tell it says native dependency Zedlib found. Yes So in this case Zedlib was available in the system So it doesn't download and build it itself. It just uses the system one and then if you compile it then it looks like this and then there's 39 compilation steps for Lua leave PNG Zedlib and this executable and and that's it and this now works works on all the platforms where the leave PNG and Zedlib and Lua are available on and This is quite interesting in itself But what can we do with this? So Now we have the same with it's time. We need to go deeper We need to all go all the way down and achieve Pythonception And ask the question Can you compile C Python with itself? Well, let's find out Here's a checkout of a C Python repository which I took from the from the git master some time ago and I wrote some of the the build definitions and There you see running the meson command and It's run the configure so you get all the things in there all checks the functions size of things all that and Then you run the build command Which looks like this saves about 143 to compiler warnings and For compared to the stuff you can make this actually kind of nice because the only thing that is visible are the compiler warnings And it's not telling you that now I'm going in this directory now I'm going in this directory I'm going in this right now I'm coming back now I'm going then and all of that goes away because you don't care what this And then if you run the compiled Python with this age Then you get the help output from the regular Python executable Which is kind of nice, but compiling Python on Linux is not actually that difficult It's it's fairly straightforward. So let's go deeper. What about Windows? So here's the screenshots of running our Windows 7 I think with visual studio LibreOffice in its infinite wisdom has decided to scale this picture down for some reason So it's a bit blurry, but there you can see it's running all the same tests And then if you run the compile Then you get this and this I don't know what those warnings are, but they don't seem to matter and Then if you try to run generated executable in the same way as earlier Then what you get is this error message Which I don't know what it is because I don't really do much Windows development is can't complaining about some missing DLL But ignoring that basically it works At Fixing the rest is left as an exercise to reader So let's look at the performance numbers of this thing Who in here knows how big the configure ac file is in? Oh, sorry, sorry Long slide so on my four-year-old I3 a 7 7 with eight cores the configured time on other tools it takes 13 seconds and And with mezen takes six seconds, which is about 50% faster build time is 31 seconds When it goes down to 16 which is again about 50% faster I I also run the tests on Raspberry Pi 2 and Configure time on that is three minutes and 11 seconds and went to one 31 which again about 50% faster And the build time is is from nine minutes to four and a half Which is again about 50% so so 50% improvement overall It's this is not entirely fair because it's not building all all of the same things, but it's roughly about the same So build definition size and so how how many of you here know how big configure ac is on python any guesses anyone So the correct answer is that is four thousand five thousand four hundred and seventy seven lines And if you haven't read it, I highly recommend that you do because it's it's both terrifying and awe-inspiring in the same Exactly the same time It has still support for stuff. You didn't even know existed and it's probably older than most people in the room And in addition to that there's the make file dot pre dots in which as you can tell is twice templated And that's seventeen hundred lines. And there's some make files in the sub trackers, but I didn't count those So in comparison the mess and bill for the same is six hundred and twenty nine lines Which is like slightly less This only includes building the core and the essential modules that are parts And this has only been tested on Linux with GCC and Windows on Visual Studio. So it was probably needs needs some work On some other ones But on the other hand, we still have about six thousand lines of spare until we reach the same same Mon of course as the first one. So I'm fairly confident with that that that's achievable All right, but can we go deeper? So here's something that I thought up as far as I can tell this is currently not possible But it's it's completely feasible to do So what if your python compilation? When something like this So you first you compile enough court code to compile the parser right and Then you compile the standard library that you have from PI files to PYC files and Then using a fairly simple script you compile those PIC files into C libraries I know sorry C arrays. So a C file and a dot eight file listing those things Right and then you compile the rest of the python and all the rest of your application that you want and Then you link the python core plus two modules plus the standard library that you built a C arrays statically and this gives you a Fully self-contained program dot exe and you can run it and doesn't need to unpack anything It doesn't need any file system access or anything It there's no possible way that it can read the wrong PYC files because it only acts it Access to them that are inside your own executable. So this would be a sub-projectable python and The things that you can do would be it's basically like freeze or pi to exe or there's a C make python Which is a bit like this But if you have it inside the build system and it's inside The the core build of the language itself then it works on all platforms and all use cases out of the box Then you can create cross-platform vacations where you embed python because it's just one dependency and it just compile it in and start call it and stuff starts happening and You can ship actual exe files and there's no file extension circus anymore One of the biggest pain points in cross-platform development in medicine that we had is the shebang line because on windows You must have the PY extension because otherwise it doesn't work on other platforms You mustn't have them because that's the distro will say you can't have that somewhere user being python points to python 2 somewhere points to python 3 and and it's like It's it's enormous mess and if the python exe itself goes away then all of these problems go away as well Your applications don't require python to be pre-installed on windows This is especially a good thing and they don't try to use the wrong version by accident You don't get weird symbol Mangling thing look up to where some of the symbols come from this deal or some come from that one And then you can combine this with GTK or with be aware which is a great project for creating a Multi-platform GUI application And then you can ship multi-platform GUI applications, which you can pile from your own source into Like full executables and just ship them and your users don't have to think about any of that Going on further so there was a presentation by I think just triplets how they're using python Where they can pilot and they boot directly into ufi well you can do that You could have bare metal support inside the python So python would be sort of like a micro less micro python where if you have a slightly bigger thing And you want to control it completely from python Go ahead you can do that And if there are people who run containers then you can do stuff like you have your Linux kernel And you the only process that's on the system is the python process Just a single process you launch that and this has interesting implications because most exploits in the world try to Open a shell on the target machine if your system doesn't have a shell at all all of these things stop working Which is kind of nice and and there's all sorts like sorts of scale things that you can do because you don't have your Images that you launch are very small so Winding down so in conclusion Python is said to become a core dependency for a significant fraction of the Linux you will land currently it hasn't been they have been using make and shell and orc and tr and said and all of those Wonderful tools that we know and some love But which ours has complete maintenance nightmares. So those are all gonna go away eventually Which is which is nice And one of the main things that comes from this is that everything will become faster so For those who don't know if you run a shell script and you do a string comparison What it actually does is that it spawns a new process every time you compare two strings and Especially on Windows with where process spawning is slow Then this like just being able to do in a process string comparisons is a massive cane and And if we have these new sorts of tools with with new sort of things then new things become possible Some of which we probably don't even realize yet So coming back to the original question was that all build systems are terrible. And why should I care? so In conclusion what I would like to say is that the reason you should care is because some build systems are actually fun exciting This has been my talk. Thank you and open for questions Well the question nobody Yes Is this a freeze thing you mentioned Just an idea or some actual project we can look at So It's this microphone on okay, okay, so This one so this is not at the moment done. It's just that it could be done In In theory by someone else, which is how these things always go But there's nothing like inherently difficult about this It's just works that needs to be done So like you need as an example if you I'm don't I'm not entirely sure if the in Python modules The entry point is always named the same if it is then obviously you can't because all of the things go in the same Same executable, so they have to have unique names, but as an example the G streamer Multimedia framework, which is like 150 plugins they have all the macro magic necessary so if you're compiling Dynamically or shared libraries, then they all get the same name And if they if you are compiling them statically then they get unique names and something like this would need to be done If there are duplicated symbols, but I'm not aware of the like actual details of whether that's possible Thank you very much for talk I'm curious how you make sure that Mason is still fast when you make a change so As said we have the the testing framework, which is in place and The biggest so currently when we run CI it takes quite a long time. So if it gets lower, we will definitely notice this Why quite quickly in fact and and one of the biggest problems that we currently have is that? Due to some bugs in other parts of the things we have to do things a bit slower But only in CI In regular use it doesn't matter. This is not an issue that comes up But for CI we have to do some fudging and because of that. It's like to work slower, but but it's still still pretty fast and We're very much counting on the fact that people are running Building their stuff against mess and master and if we get slower, they start yelling and then we'll fix it Okay, great. Thank you very much Do you have another another question? Yes, so you talked about Compiling for for Windows for Mac OS X and so on and so on consider I have to build a product and it's an embedded product and I have some Microcontroller can I use Mason to to work with some not yet known compiler and would you recommend it? So currently we only support the compilers there, so that's GCC clang ICC and then like 10 different for drunken pilers with some guy contributed And we don't you can't compile on a compiler. That's not specifically supported because we need to know how you run that But if you want to add this we are open for patches We love patches if you if you send in and it's not actually that difficult So the the language support for D was added by one guy who had never seen the code base before in about two days And just adding a C compiler An existing C compiler would probably not be that much for Joe So that means that you Have predefined rules for compiling what if I in a project I want to have my dedicated Options so I want to change Optimization levels and what else is possible? Is that is it possible with with mason or is it impossible because the rules are already closed So what we do is that this predefined? target built optimization types, so there's Just debug this optimized for speed release and minimum size, and then there's the Nothing at all, and then you can specify your own compiler flags and We just use those and we don't use any of our own. So if you want to override, that's fine another question No, thank you so much The next talk will be done by Nicola la Rossi in some minutes Thank you