 Okay, so we have Alan Martin speaking about how to build C++ applications with Python as the core Kind of so please introduce him and Big applause for him Hi everybody, I'm Alan and I work as a software engineer at Ableton in Berlin, Germany Let me introduce you to what we do At Ableton we make live push and link Software and hardware for music creation and performance First we make live, but you can see on the laptop screen here Live is a digital audio workstation for creating musical ideas turning them into songs and performing on stage On a technical level, it's a C++ desktop application built for Mac and Windows from a pretty old codebase 16 year old Then we make push Push is a USB device which becomes a powerful Electronic music instrument when combined with live it embeds Multicolor display at the top that renders a cute quick scene powered by cute, which is a C++ application framework And then we make link Link is a technology that keeps music applications and devices in sync by synchronizing their beat It is available to app developers as an iOS SDK Now you might ask, but what does that have to do with Python? Well, we do use Python at Ableton First we use it to write remote scripts for MIDI controllers these MIDI controllers are Devices like push that musicians can use to interact with live in order to enhance Their music creation workflow and to perform Then we use Python scripts to automate continuous integration and releasing new versions of live and Last but not least since it's the topic of this talk We are using Python to build and test C++ applications such as live So let's see how we are building live with Python In order to build C++ applications You have to compile C++ source files and sometimes also see an objective see source files And so you compile them into object files binaries Which then are linked by the linker into libraries and executables When building live more than 1400 files are compiled to product 88 libraries and eight executables Fortunately, there are tools to help you Build C++ applications, so you don't have to call the compiler and the linker manually yourself These tools are called build systems and you might already know the ones I've listed here Unfortunately most build system tools Support only one platform and we want to build live on Mac and Windows and actually Live used to be built from Mac files on Mac and visual studio projects on Windows and it was a total maintenance nightmare So we started using a higher level tool that would abstract most of the platform differences This tool is JIP JIP stands for generate your projects and it's a meta build system a bit like CMake and Qmake It's a multi-platform tool that generates files for platform specific build systems It can generate Mac files, ninja build files, visual studio projects and Xcode projects And it is written in Python Nowadays it is mostly used to build Chromium Electron which is the base of GitHub's Atom text editor Node.js and the V8 JavaScript engine And it is also integrated in the live repository to actually build live Here is the live git repository very minimalistic view with just what matters for that talk and At Ableton we use git some modules to integrate third-party code Or to reuse our own code across several projects and here in order to build and test live The live repository contains build system, which is also an Ableton repository as a Git submodule And there's also a root-level JIP file named live.jip Now let's see the content of build system So build system itself is a git repository And it contains first of all the JIP repository as a submodule So we have some kind of submodule exception happening here, but it's manageable Then it contains Python scripts To build and test C++ applications. There are three top-level scripts Configure.py is responsible for calling JIP with the right arguments Then build.py calls the correct build system to either MS build, ninja or Xcode build And then run.py calls C++ test runners or Qt specific tools Going into the full details of how live is built would take way too long for this talk So we're just going to use a simple hello world example to see how the build system scripts do their work So here's the hello world project really Simple for now. It obviously contains a copy of the build system repository otherwise that wouldn't be the build system hello world And like in the live repository it has a top-level JIP file The C++ code to compile is spread between a source folder and a test folder Which also contains a copy of catch, which is a C++ unit test framework Now let's have a look at hello world.jip Here two targets are declared hello world and hello world test They both will produce one executable build from one source file So pretty simple, pretty easy You might find the JIP syntax familiar and That would make sense because well, it's a Python dictionary actually The content of hello world.ccpp is straightforward classic C++ hello world And in test hello world.cpp we use the catch test framework to just write this really dummy unit test that always passes Now let's build that So to build and test the hello world project we first call the configure script it indicates What it's doing and for which configuration? So win is the platform it could be Mac, but I'm running on Windows 64 stands for 64-bit could be 32 and DLL is the linking style. It could be static Configure.py is as I said responsible for calling JIP which loads the hello world.jip file and generate Visual Studio files since we are running this example on Windows The generated files end up Under the id folder in a sub folder which matches the configuration In a second time we call the build script Based on the configuration it prints what is about to happen Then it calls MS build since it's the default on Windows a bit repeating myself You can see in the MS build console output that two executables have been built Hello hello world.exe and hello world text.exe Build.py finally prints what happened Which is really convenient when there are dozens of targets with hundreds of files and you don't want to scroll Everything like when we're building live The build artifacts end up under the output folder Here again in a sub folder based on the configuration there's one sub folder more Called debug because when calling build.py we can choose between debug and release This division into sub folders based on the configuration allows us to Simply build and test several configurations of the same application in the same repository And now we can run the hello world executable Hello world To finish that example Oh, sorry We call the run script with the cpp test command And it will basically call every c++ test runner from the correct output folder By convention only executables ending with tests are treated as c++ test runners Thus hello world.exe won't run here only the hello world test.exe Run.py also prints a summary of which test runners have passed and which have failed Which is again pretty convenient when you have dozens of them and you don't want to read all the output So now we've seen how build system works in a very small project But now let's talk about what it means to use it with complex applications such as live So compared to the hello world project, there are a lot more files in the library repository so it takes Much more time like really much more time to build and test But it works in the exact same way The only thing to do is simply configure build and run nothing more But don't get fooled by the appearance simplicity The fact that there are only three scripts to call Doesn't guarantee on itself that these scripts are easy to use maintain and extend in fact these scripts could have been developed in a totally chaotic way and It would be a mess Not only for developers who work directly on them But also for developers who are using them to build and test their applications And developers really don't like to work with a mess So, I mean you may have guessed we didn't develop our build system in a chaotic way. Thanks Otherwise, I wouldn't be giving a talk Would still be in Berlin hiding from my angry colleagues So no everything's fine instead of chaos. We embraced consistency The three top-level scripts configure build and run share a common architecture They only use modules from the Python standard library principally our class logging and sub process and They follow the same sequence of actions first the past the arguments Including finding the project jib file since it's the entry point Then they figure out what the users the user wants to do and then they just do it More than just a common architecture these three scripts Actually share a common design this design is made of design principles and by implementing these principles the scripts become easier to use Maintain and extend now right now. I want to share with you three of these design principles They are generic so you should be able to apply them to your own strips First principle fail early loud and clear I don't really think I have to explain why but we'll go for an example just to be sure Here is a small script We declare an argument buzzer with one argument target dear Then we get the target directory actually out of the arguments and We count how many entries they are in that directory and We print it on the console When running the script on the hello world project, they are six entries Everything's fine Now let's try to do the same thing on a non-existing directory Well exception So we get an error which is actually a platform specific error. It's a Windows error So my colleague working on Mac won't even see the same thing Here it's pretty easy to trace back where that error comes from then they are not so many lines in that script and It's pretty obvious what happened, but if you have a really complex script, which does a lot of things It's really helpful to explain to the user. What is wrong is what you type on the common line It's not a bug in the script So how do we do that in Python? Well, the trick is to check the argument type as early as possible So here I have a function which Basically takes a path figures out whether that path is the directory if it's not it raises an argument type error exception with a comprehensive message Otherwise, it just returns the path and we use that function in the type of the argument So now when passing the arguments the arch parser will actually call that function and if Everything is fine Nothing changes still the same output the user sees the result but if The folder doesn't exist Then we actually get an error message which is clear because now we know it's something wrong When passing the arguments and not a bug in the script So fail early loud and clear just to make your scripts easier to use The second design principle is support custom defaults, so you might be wondering custom defaults Don't really go together. Well, let me explain I've showed you that Jeep can also generate ninja build files Ninja is not the default build tools on Windows and Mac, but we support it because it's pretty fast So if I want to configure the halo word project with ninja, then I call configure the configure script with minus minus ninja I get The output with extra ninja Indication then I call the build script with minus minus ninja and I get the ninja output And then I can run the halo word and everything is the same. It's the same C++ code and The compiler calls should be identical But I'm a lazy man. I don't want to tap minus minus ninja every time I call configure or build and I do that a lot every day So to solve that issue We've created a way to have custom defaults So in a file, which again is a Python dictionary Users can write what they want to be given to the arc parser as the first argument So these arguments go at the beginning and then whatever they put on the common line goes next So they can always on the common line override anything that is in the dot RC file So There's one dot RC file per user in the user folder Then they can they can be one RC file per Build system repository. So when you have several working copies, it can be useful to have different settings based on the working copy So this file is getting ignored. It's not checked in the build system repository And then as I said common line arguments at the end Text precedence since the arc parser reads the arguments from left to right so now with the content of the RC file Including minus minus ninja when calling configure dot pi and build dot pi. I can just call configure and Call build and I don't have to type anything else Now the third reason principle do not integrate project specific features So the build system repository is completely Live agnostic. It doesn't know that it's used by life It can be used actually to build any C++ applications If we have a look at the build system repository and how it's composed We saw there's jip there are the three main scripts and to Share code between the three main scripts base a Python module Which contains functions for Argument passing like the existing deal function. I've just shown functions for getting the default arguments based on the platform and Function to figure out where is the top-level jip file So there are some heuristics. So As you as you've seen I don't have to Specify where is the halo world of jip or the live the jip file when calling configure and build So now for this example, let's imagine there is a fourth script Because configure build and run are pretty big pretty complex So let's just go with a simple arg parser Here is the content of the simple arc parser We import arc pass. Obviously We import constants from build system but then we can use to define to Specify the default value of the first argument, which is the platform Then we have the word size and then we have the linking And this script only prints the content of the argument namespace nothing fancy So let's call it just to see how it looks Here we go just prints Linking is DLL platform is win and what has a 64 since these are the default values Now let's go back to the halo world project In the halo world project. I would like to call this script with I don't know a Specific argument that I really would like to have So here we could imagine But we want the halo world to Change the language. So instead of saying halo world it would say bonjour le monde or hola el mundo or whatever So we give the language on the common line Obviously that won't work for now. There's no such argument so now I might Be a bit sneaky and ask can we add that to build system, please No We do not integrate project specific features ever because it adds maintenance cost and It leads to API breaking changes as soon as another project starts using that feature and then the original projects Decides to change to change it. So no instead We make it easy to write script extensions So here we're back to the simple arc parser script with some space because I'm going to fill the blanks first thing we do is include another Module from the build system Then we create the script extension I Will come back to that part In a few slides, but basically we create the extension and then Afterwards we use it So we have a function called extend arc parser before Which we put before defining the other arguments and another function called extend arc parser after that we put after Defining the arguments So this is the only thing we have to write in the build system script Now, let's see how it works in the top level projects The first thing they have to do So the first thing hello world has to do here is Create a Python module named build system extensions. That's the convention It has to be like this and in that Module They can there should be one file at least one out called simple arc parser And this file matches the name of the extension that was created Here we define only one of the two functions So here we define the extend arc parser after and now when we call the script with our specific argument It works No errors So how does that black magic happen? Well first as I told you there is a A script extension Python module and In that Python module there is a function called get script extension When calling that function we pass two things the first thing to pass is the script name and The minimal API So the minimal API will define what are the functions that the extension has and Now we go into the dirty bits So who has ever done the first line import I am P Okay, let's maintain So this is the low-level way of importing a Python module The first thing you have to do is to find it like Sorry Using a collect a list of paths Where that module could be and the name it has Then if it has been found you can actually load it Then there's a bit more craft to handle closing files and what not but what you have to keep in mind here Is that now we have a function called find and load? Find and load module and this is now really useful to create the script extension So creating the script in sanction for real So we're back to describing that get script extension function takes two arguments the script name and the minimal API So the first thing to do is to find The build system extension module so the one I told you this is by convention It's how it works and we basically do that by just going up on the tree structure And listing all the parent So we start at the hail world Repository and then just go up up up up up until the root of the drive And if in any of these folders there is a folder named Bill system extension actually not just a folder a Python module name bill system extensions then it is loaded and If we have found it then we can try to find inside it a module which has this desired script name if Nothing has been found that can happen Because maybe the user the halo a project or another project hasn't Defined any extension This script extension is just a dummy empty object and Then we iterate over each attribute of the minimal API and if The script extension doesn't have that attribute yet. We just set it so that way The script extension that is written by this function is Complete so the script that is using this object doesn't have to check whether the attributes exist and It can just rely on this minimal API being complete and Yeah, just work to wrap it up We're building live with Python To echo the title of the the talk Python is indeed the keystone since Without Python nobody can work on life About the design principles The thing to remember is that having some is good it helps you write better code But knowing and sharing them is better. So I invite you to come to me and to your fellow Europe by finista and share whatever design principles you are using in your code and in your scripts to make them better So then all together we can make the users of our scripts happy Thank you for attention We actually have like a lot of time for questions. So raise hands, please Yeah, so just a small question. I was wondering why would you why did you decide to include the Debal system as as a model instead of packaging and deploying it We have kind of a history at Ableton of Having everything in the repository. I mean almost everything. So we obviously don't have the C++ compiler. We don't have The Python interpreter itself, but all the code we run we compile and we use is inside The live repository. So I think the some module So so then that means that if you want to pick something like a hot fix or anything for your libraries You actually have to update your submodel, right? Yes How is the experience going that way? I mean, I'm not I'm not saying that's bad No, no, it's fine. We have done Way more acrobatic things with some modules Hands up, please someone else Did you think about using waf on score scones instead of building your own build system? So, yeah, so Python based. Yeah, I know So the thing is waf and scones Call the compiler directly But we are really attached to Using certain IDs like Xcode. I know that Mac developers really like to use that tool and Unfortunately, Xcode only works if you have an actual Xcode project So Jeep being able to generate project files that are really look alike that really look like the native projects was a big plus and Then the build system repository is merely a wrapper around Jeep more questions The talk is being recorded. So Done the question, but did you look at CMake? Yes Yes, we looked at CMake and we actually have a colleague who is one of the main maintainers of CMake The thing is that when the transition was made from the Mac files and visual studio that were checked in the library repository to Jeep at that time CMake was still pretty annoying and But nowadays it's a viable option Let's say more questions I have one myself How do you actually do do you support dynamic loading like you have gone through the packages and going up like a recursion the directories to the parents Looking for modules, but how do you actually set up the order in which they are executed? So here basically the all parent folders function returns a list or No, I think it's even it's a generator, but anyway, it's So the list of the current folder and then its parent folder and then its parent folder until the root and just because of the order in which the paths are Yep, we have to go back here The find module function which takes all this path just does it for you So you just have a one extension. You can only look for one file. Yeah, so the first much build system extensions Python module that matches is the one loaded and that's the default behavior of Python for any kind of module Have you actually looked into them with setup tools you have like entry points and you can Register like extensives if you want to have a look they If you have seen track for example or by test they load the functions like setting up extensions points You can actually extend through those points and yeah more than I make although it doesn't look like you need it Yeah, actually in the end we were pretty happy about the number of lines. It is it's It is small Okay, please Thank you