 Good morning. My name is Thomas. And today, I would like to share some of my knowledge about cross-compilation tool chains. I'm kind of surprised by the number of folks in the audience for such a topic. But, well, that's interesting. So I work at four electrons acting as the CTO of this consulting company. I mainly work on the canal daytime and on build-root nighttime. So build-root is an embedded Linux build system. And part of the knowledge for this talk comes from this experience working on this tool and happened to live in the southwest of France in the sunny city of Toulouse. And on the right-hand side of the slide here, you can see a drawing from a very nice artist who draws sketches from speakers and audience members at the Canal Recipes Conference, which takes place every year in September in Paris. And it took place two weeks ago. I think it's a nice conference. So if you've never had the chance to attend, I recommend attending it. They are very, very good speakers. It's kind of the shameless plug for this nice conference. But to give you a chance to leave the room, in case you're not really interested, I wanted to just make sure that we're on the same page on what this talk is going to be about. I'm not a tool chain developer. I work on a build system with many other people. So I do have some amount of knowledge about compilation, cross-compilation issues. But I'm definitely not a GCC or Binadil's or C-Library developer, by all means. It's really some experience gained by building simple tool chains for an embedded Linux build system. So this talk is really an introduction type of talk and not in-depth type of talk. So if you already have some knowledge about tool chains, feel free to just leave the room right now. I won't be surprised by that. So we're going to focus on simple GCC-based tool chain. We're not going to cover LLVM. We're not going to cover advanced use cases like link time optimization, graphic optimizations, all that things like that. We're really going to stay in the really basic cases. I see nobody left. So hopefully, all of you are going to be interested. But who I follow in my talk? So what is a cross-compilation tool chain? That's obviously the first question to ask. So it's a set of tools that allows you to build source into binary code for a target platform that's different from the one you're currently running on, the one on which the build process is operating. Most of the time, people think of it as different CPU architecture. But it can also be different operating system, different C library, different ABI, and things like that. So for example, if you're on X86, Linux X86, and you want to build an X86 binary for Windows, you're doing cross-compilation, even though it's the same CPU architecture. If you're on, let's say, building on ARM, so your machine is an ARM platform that's using a given ABI, and you want to target a different ABI, it's also cross-compilation. In this build process, three machines are involved. The build machine, which is where the build takes place. The host machine, which is where the execution takes place. And the token machine for which the programs might be generating code. So in most Linux distributions, you have pre-built native toolchain, which is the one you use to build application that you run on the same machine. And this toolchain has been built with build equal host, equal target. For example, you have an X86 machine. It has been built on X86. It runs on X86 and provides and generates binaries that runs on X86. That's a native toolchain case. The cross-compilation case is where build is the same as host, but different from the targets. So for example, you are building a toolchain on X86, running it on X86, but it produces binaries for a different CPU architecture, or different operating system, or different ABI. So the target is different. Those three machines, they directly correspond to the three arguments of all auto-configure scripts that have the same name, build, host, and targets. So by default, when you run a configure script, it automatically guesses that you want to do native build. So when you do cross-compilation, you need to override these to specify what you want to do exactly. So the auto-con specifies a concept of what they call system definitions that are represented by tuples. So a system definition describes a system. It defines its CPU architecture, its C library, its operating system, its vendor, its ABI, and sometimes a bunch of other information. It takes different forms. The full form is the one that you can see on the slide. It's arch, vendor, OS, and then some combination of information about the C library and the ABI. There are shorter forms that, for example, avoid the vendor part. So the different components that you have are relatively easy. So arch is the definition of the CPU architecture. It can be R, MIPS, X86, and a bunch of others. The vendor is mostly a free form string. It's just a string that can identify the company or the entity that produced the tool chain. There are just a few corner cases where it actually means something. But for most situations, it's just a free form string. The OS is the operating system you are targeting. So for the case of this talk, we're only going to consider the cases non and lonex. And I'm going to explain in a few slides what they mean. But there are, of course, other possible operating systems. And then libcABI, as the name suggests, gives details on which C library you're using and which ABI you're using. And we're also going to talk about C libraries and ABI in the rest of that talk. So here are some examples. ARM Foo Non-EABI is a tool chain tuple that indicates that it's a bare metal tool chain. So the non part says it's a bare metal tool chain that targets the ARM architecture coming from a given vendor and using the ABI eABI. ARM, unknown Linux GNU ABI HF, it's a Linux tool chain as the OS part of the tuple suggests. It comes from some unknown vendor, targets the ARM architecture, uses the Jellipsy C library and the eABI HF ABI. So all those information are encoded in the tuple. And it goes on and on like with the ARM, big Indian, Linux targeting, Usylipsy using eABI tuple, which is the third one here, or another one for the MIPS architecture that you see on the slide. So the main two values of operating system we're interested in in this talk are known in Linux. And you saw some example of those in the previous slides. So non is used for bare metal tool chains. So they are used for development without an operating system. So directly on the metal as the name suggests. The C library in use in such cases is generally newly. So it provides C library services that do not require an operating system like streak manipulation and compare and other basic services that do not require an operating system beyond it. And then optionally in new live you can enable a number of basic kind of system calls by implementing hardware specific operations or connecting that to your small real-time operating system if you have one. But it's mainly meant for development without any operating system. So you can use such tool chains in the Linux context to build your bootloader, to build your Linux channel but you cannot use that to use a space Linux code. The second type of tool chain are Linux tool chains. So they are used to develop Linux user space applications and share libraries. So their main difference with bare metal tool chains is that they contain a full feature C library that integrates with an operating system in that case a Linux channel. There are different C libraries that are available and we're going to talk more about that in the next slides as well. In that case the C library supports Linux system calls. So when you're creating a network socket when you're opening a file or anything like that the C library talks to the channel to get access to this service. As opposed to bare metal where no such services are available. So such tool chains can be used to build Linux user space code but they can also be used because there are kind of a more or less of a super set of bare metal tool chains. They can also be used to build your bootloader or your channel. So if you've already building your channel or your bootloader you may be using a Linux capable tool chain even though the C library is not being used it just works fine. So what do we have in a cross compilation tool chain? We essentially have four main components. So it's not that complicated in the end. If you dive a little bit into it it's not that bad. So we have four main components. The binitals, GCC, the Linux channel headers and the C libraries. And that's pretty much all what we need to build a tool chain from scratch. So a few dependencies are needed to build GCC. I'm going to cover that as well but they are kind of pretty easy to build and to use libraries. So let's cover those components one by one. The binitals provided by the group project and they define it as a collection of binary tools. You have two main tools. Yes, which is the assembler. It turns assembly code written in text format into binary code. And you have the linker LD which takes a bunch of object files and combines them into another object file or a share library or an executable. So that, those are obviously the two most important tools from the binitals. And it also provides a number of other like debugging analysis or other kind of tools to analyze binaries. So if you have never used like Redialif, Objectum, stuff like that it's very, very useful. Tools for stripping binaries and so on and so forth. But those are not that important for the completion process by itself. LD and AS are by far the most important ones. The binitals, it needs to be configured for each CPU architecture. So if you have a native X86 binitals on your system which you probably have as part of your Linux distro, it cannot by itself build or manipulate arm binaries. So it can be configured to, you can configure binitals to support multiple CPU architectures. But by default it's generally built with just targeting one CPU architecture. So that's why it's not as simple as using your native binitals and passing some arguments to say, hey, you build some arm stuff or build some poor PC stuff. Generally need to build separate binitals. Building binitals is pretty easy. It's just other tools based. So it has configure script to run it. The only thing you need to say is what you're targeting. So here we find again the build host targets. Well parameters. I've left out build and host because they just let autoconf guess what my build machine is and my host machine is because I wanna build on my laptop and run on my laptop. It's the only thing I wanna override is for which architecture I'm going to be generating binaries. And in that case I'm going to generate for our Linux GNU ABI HF system. Another thing that you need to pass is the SysRoot. It's a directory that contains stuff. For now I leave it to that because I'm going to talk more about the SysRoot a little bit later. The next component is obviously GCC. It's by far the one that is kind of your entry point into the tool chain and that the one you manipulate directly. So it's the GNU compiler collection project from GNU. It has frontends for many source languages. C, C++, Ada, Fortran and others. I just read that they got rid of the Java support in recent times. It has backends for many CPU architectures. So it essentially converts the source frontends. They convert the source code into an intermediate representation. It applies a huge number of optimization passes on these intermediate representations. And then the backends, they take this intermediate representation and convert it into an assembly code for a specific CPU architecture. So it does not generate by itself binary code. It only generates text assembly code. And AS needs to do the rest of the work. It provides different things. It provides compilers themselves. So CC1 for C, CC1 plus for C++ and it has others for if you support other languages. But those are not the ones you normally use directly because GCC is in fact the second item in the list here. In fact provides what they call compiler drivers. So kind of a wrapper program that will orchestrate the different steps of the build. So when you run GCC or when you run G++ behind the doors, what it's doing is that it's calling the actual compiler itself. So CC1 or CC1 plus. It will be calling the assembler. It will be calling the linker for you and doing all the work for you. So GCC and G++ are not the compilers themselves that just drive the compilation process. And then you should of course always use them. Another thing that GCC provides and that some people are not necessarily aware of is that it provides runtime libraries as well. So it is the compiler itself but it also provides a bunch of libraries that you need on your target to be able to run these programs. And they are different part of the GCC runtime, libGCC itself, libSTD C++ which is the standard C++ library and other runtime for other languages. So like Fortran has its own runtime and other languages may have their own runtime as well. And the last thing that GCC provides is the set of headers for the standard C++ library. So GCC is providing not only the compiler but also runtime libraries for various languages. Building GCC is a bit more involved than building binitails. It's not like a single configure line and off you go. There's a two step process that we're going to cover a little bit later to build GCC in a cross compilation situation. The next component in any cross compilation tool chain are the canal headers. So the one ultimately to build is a C library but our C library wants to communicate with our Linux channel. And in order to communicate with our Linux channel needs to know the system called numbers, the various data structures, various definitions and things like that that actually come from the canal. And instead of duplicating these information in the C library they are kind of shared and extracted from the canal. So in the canal source code there's a split in the headers between user space visible headers and internal canal headers. And this split has been done like a few years ago I think it used to be kind of a mess and nowadays it's much better organized in the canal sources. So there are UAPI directories in include Linux and other places in the canal where UAPI stands for user space API. So those are the headers that are visible to user space and other canal headers are not exposed to user space. There's a special target in the canal build system called headers install that will take all those UAPI headers from the generic ones plus the one for your specific architectures and a bunch of other things. And that will be messaging them a little bit. There are a few things that need to be adjusted in them and then install them into a location. So it's just installing a bunch of header files that describe the canal system calls, canal data structures, a number of definitions and so on. As of Linux 4.8 it installed about 700 header files so it's quite a bunch of header files installed here. One question that comes up pretty often when people are building or using existing tool chains is I'm running a given canal version of my system like let's say a Linux 4.7 and I have a tool chain that uses canal headers that was built like I don't know, two years ago it's using to canal headers 3.10, is that the problem? Or the output is I have a system that's, I don't know, running 3.10 and I have a very modern tool chain that's using canal headers 4.4, is that the problem? So the canal to user space ABI is backward compatible so the canal developers are normally never removing system calls or changing the semantics of existing system calls. They are only extending this API with potentially more flags, more system calls, more features, but they are not changing what already exists. So the consequence of that is as long as the headers that have been used to produce your cross compilation tool chain are at least as old as the canal you're actually running in the target, you're fine. If you're using a tool chain that has header that are more recent than the canal you actually run, then your C library will think that your canal will support system call A, B, or C that is very new, but your canal will not actually provide them and that will not work. So when you're choosing a tool chain or building your tool chain, just need to make sure that you have headers that are at least as old as the canal that you intend to run on your system. You don't need to have the exact same version. So back to the example I had, if you run 4.7 and that's the canal you actually build and run on your target, but your tool chain has been built with canal headers 3.10, that's just fine. It will work fine, all right? So you don't need to have an exact match there. I just want to know the version of the canal headers that are in use in your tool chain. The Linux canal has a file called Linux-version.h and it contains this Linux version code definition that is a bit cryptic to read, but it actually contains the three digits of the canal version, like in that case, it's 3.13.0, so it's pretty cryptic to get from 199.36, but you get the magic formula just below, so it's not that a big deal to infer that. The next component is the C-Library. So it provides the implementation of, well, the POSIX standard and very often a number of extensions that are either Linux-specific or sometimes even C-Library-specific. They are, for the ones that we really care about are based on Linux-system calls and there are several implementations available. Jellipsy, UslipsyNG, Muscle, Bionic, which is used for Android systems and a few other more special purpose, so I already covered NewLib, which is mainly Bermittal, so this one is kind of an exception, it's not based on Linux-system calls and that Lipsy or K-Lipsy that are really like tiny, very, very tiny C-Library that cannot really be used to build, I would say, full-featured user space applications but more for very minimal Linux systems. So I'm mainly going to cover Jellipsy, Uslipsy, and Muscle as the three most important C-Libraries that can be used to build a full-featured Linux user space. So before we get into the details about these two three C-Libraries, what they provide, each of them provide the Dynamic Linker. So the Dynamic Linker is the program that actually gets triggered when you run a dynamically linked application. It's not actually your application that starts up first, it's the Dynamic Linker, which is responsible for mapping in memory the shared libraries that your application is using. Of course, that's only for a dynamically linked application. If you have a statically linked application, no dynamic linker is involved. That's provided by the C-Library. It's the C-Library, of course, provide, well, the C-Library itself, libc.so, and sometimes a bunch of companion libraries, libm, librt, libpthread, and so on, providing other parts of the POSIX standard functions. And obviously it provides the standard C-Library headers, stdauto.h, and string.h, stdauto.h, and so on and so forth, probably well aware of the POSIX interface. So the three C-Libraries that are, I would say, widely used today on Linux systems, are first, the Gealipsy. That's the de facto standard. Every Linux desktop or server machine is using Gealipsy, provided by the GNU project, started in the 80s. That's the full-featured C-Library. Supports many architectures, many operating systems. For the folks doing embedded, Gealipsy has no support for no MMU architecture. So if you're doing ARM Cortex-M, or if you're doing, I would say, Blackfin, or MicroBlaze, no MMU, or those kind of no MMU architectures, that Gealipsy is just not a possible choice. There is no official support for static linking. It kind of works in some specific situations, but the Gealipsy developers explicitly say that it's not supported. So dynamic linking is kind of the only supported way. It does provide ABI Backward Compatibility. So what this means is that if you build and link, of course, dynamically, a program against Gealipsy, you can hear from no upgrade Gealipsy to a newer version of Gealipsy. Without relinking your application, it will continue to work. So that's kind of helpful, obviously, for binary distributions, because they can upgrade the C-Library without necessarily rebuilding the entire world that sits on top of it. There's almost no configurability, so it's just you build this Gealipsy, you have all of it or none of it. And it used to be too big for embedded, so that's why some of the alternatives that we're going to talk about exist. But whether it's still the case, it really depends on your situation. Even ARM systems have gotten bigger and bigger and bigger and have more flash, more RAM, and so on. So what used to be relevant alternatives 10 years ago may not necessarily be very useful anymore, so lots of people are using Gealipsy and embedded systems, that's perfectly fine. It's licensed under LGPL v2.1 or later, which is kind of an important criteria. There are other licensing options for other C-Libraries. The other one that is also usually quite well known is Usylipsy or it's now called Usylipsy-NG. It's been started in year 2000 and the goal was to provide a smaller C-Library. At the time, embedded systems had like eight megs of flash, 16 megs of flash, so the size really mattered. And as I said, nowadays many embedded systems have much higher capacity, so it's not necessarily as useful as it used to be, but people still work on systems that have such low amount of flash, and in that case, Usylipsy or some other possibilities are still interesting to have. It provides a high level way of configuring the C-Library, so compared to Gealipsy, that's just all or nothing. In Usylipsy, you can select what features you want, you want network support, you want IPv6 support, you want thread support, you want locally support, and so on and so forth. So you can fine tune the C-Library to whatever you need. It supports many architectures, including some not necessarily supported in Gealipsy. Doing a Usylipsy port is often easier than doing a Gealipsy port, so some of the people doing Usylipy architectures, they very often start by a Usylipsy port as that, yeah, that's an easier target. Compared to Gealipsy, however, it only targets Linux as an operating system, so it's kind of a narrow focus. They don't provide any ABI backward compatibility, so if you upgrade Usylipsy to a newer version, you may need to rebuild all the libraries and applications that rely on it, which for most embedded system can be fine, but not necessarily for binary distributions. It supports a number of no-MMU architectures, or no-MMU, Blackfin, and others that I already mentioned, so that's one figure that Gealipsy does not have at all. Some people think that it's related to Usylinux in some way because of the name, but it's not anymore related to Usylinux, and Usylinux itself doesn't really exist anymore, so Usylipsy is just yet another C library that you can use on top of the Linux scale. It's not like Gealipsy is for Linux and Usylipsy is for Usylinux, that's completely wrong. Usylipsy is used on top of the regular Linux scale on ARM, X86, MIPS system with MMU support, so it's completely unrelated, so don't be confused by the similar naming. There's explicit support for static linking, it's one of the original goal of Usylipsy to support static linking and produce relatively small binaries even when doing static linking. So the original Usylipsy project is kind of dead, there has been no release since May 2012, and but since I would say last year or a year and a half or something like that, I don't know if Waldemar is in the room. Is Waldemar in the room, not in the room? But he's here at the conference, so Waldemar BroadCorp, I'm hoping I'm pronouncing it correctly, took over the project and forked it under Usylipsy NG, and since then he has been doing, I think, 18 or 19 releases over a year or a year and a half, so the project is active again. I think he has merged new CPU architectures, dropped some old that were obsolete and not working, so the project is no kicking again, and that's very nice because it used to be really a pain to work with a dead upstream project, nowadays it's really working again. It's licensed LGPL V2.1, pretty much like Jolipsy, except it doesn't have the all later statement. The last one that is interesting is muscle, so it started much more recently, only 2011, and one key difference is that it's MIT licensed, so some people are getting interested in the project just because of its licensing. It is under very active development, the mailing list is very, very busy. There's support for a fairly large range of CPU architectures already, and more is being added release after release. They recently added no MMU support for CPU H2, specifically for the Gcore open source processor. There was a talk at last ELC in the US about this processor, so you wanna check out the videos, it was an interesting one. There's no configurability, but it's still small, even currently smaller than Jolipsy, and especially for statically linked scenarios, it's pretty, pretty good. They try to conform to standards in a very, very strict way, sometimes even stricter than Jolipsy and Jolipsy, which can cause a few build issues with user space libraries and applications that you may have, and they have a very, very nice page comparing the three C libraries in much more details than I can do in such a talk, so there is this very nice table with list of features and whether Jolipsy supports it, USolipsy supports it, Muscle supports it, so it's a very good way of kind of comparing the different options here. In terms of size, I did a quick tool chain build with build routes using Jolipsy, USolipsy, and Muscle, so I was targeting an ARM Therm2 architecture, and I tested those three C libraries, so you can see that the C library itself and all its companion libraries in the dynamic linker, in the case of Jolipsy, where it's about 1.7 megs, and USolipsy and Muscle are more around alpha meg for the full thing, so when I say Muscle as not available, not available, it's not exactly not available, it's not applicable, Muscle as a principle that everything is inside lipsy.so, even the dynamic linker is lipsy.so itself, so there's just a single file that you need to install and it contains all the functions, so it's not that thread is not available, it's not at all, it's directly in lipsy.so itself. USolipsy migrated to this model as well in the latest release 1.0.18, which wasn't used at the time I did the measurement, so those are based on 1.0.17, which is just a release before that, but USolipsy is moving to the same model, just to give a rough idea. So if you have 256 megabytes of flash, of course, the size doesn't matter at all and you could just as well use G-Lipsy, that's fine. If you have 16 megs of flash, a meg, meg.2 of saving starts to be a little bit even interesting. Back to other components of the tool chain that you need. There are a number of mathematic libraries that are needed to build GCC. It's pretty arcane and weird stuff, but it's anyway needed to build GCC, so you need to three libraries, MPFR, GMP and MPC. So GMP is just a dependency of MPFR. MPFR and MPC are used by GCC to make calculations on floating point numbers and complex numbers at build time. So if your program or library is having as a number of constants, floating point constants or complex constants with mathematical operations, GCC is evaluating them at build time and to do that it needs arbitrary precision libraries to help doing those calculations. So that's the, as my firm are understanding, the only reason why they are used. So it's kind of a very specific usage of those libraries but they are nonetheless absolutely needed to build GCC. So they are needed on the host machine. They are only needed to run GCC itself. So you need to have them on your host machine but they are definitely not needed on the target. So the overall build process to build a toolchain is not that complicated. A lot of people think it's like something completely crazy. It's not that crazy. You just need to build binitiles. So it's just a one configure, make, make, install kind of thing with not that many argument to the configure script. Then you need to build dependencies of GCC, MPFR, GMP and MPC. And those are just built for the host. So it's native builds, configure script, configure, make, make, install for each of them. Pretty easy. Then you install the canada headers and you've seen previously is just one single comment to say, hey, please install the canada headers for this architecture and that's all you need to do. That's also pretty easy. And then you can start the biggest part of the toolchain build. You need to build a first stage GCC and it's first stage GCC. We build a GCC that has no C library support that has only support for starting linking and it's only support for the C language and it's kind of a limited GCC that we put kind of on the side. And we use this first stage GCC to build the C library. Obviously the C library needs to be built for the target so we need to have a cross-compiler to build it. So we use this first stage GCC and once we have the C library produced it's dynamic linker and everything else. Then we can build the final GCC which needs the C library to already be built because obviously GCC is going to generate binaries, use binaries, encode the path to the dynamic linker. It encodes the shared library names for the C library, for the P thread libraries and so on and so forth so it really needs to know which C library is being used. So the C library unlike all other libraries that you might build later on for crypto or network or graphics or whatever is really an integral part of the toolchain. It's not something that's separate from the toolchain. So if you want to change the C library you need to recreate a new toolchain. From these steps, the first three, these steps look kind of sequential but there's no interdependency between the first three steps. You can build binaries, MPFR, GMP, MPC, install the cana headers in whichever order you want. You can even do that in parallel if you want. There is just that binaries, the GCC dependencies are needed to start building GCC, the first stage GCC and the cana headers are only needed at the time you start building the C library. It's just to kind of make it clear what the dependencies are. And maybe this, I don't know if it's really readable, makes it a little bit even clearer. So that's the dependency chain of the packages and build route that produce a cross compilation toolchain based on G-Lypsy. So it's a dependency chain. So the build order is the reverse of the one you can see here. So the build actually start from there and goes up in this direction. So to build a host GCC initial, we need host binutils, we need host MPC and so on and so forth to build G-Lypsy. We need host GCC initial and the cana headers and to build all GCC final, we need G-Lypsy to be built. But that's pretty much all what you need to build a pretty simple and working toolchain. And with that, we build a large ambivalent system. You can build a QTE, X.org, whatever, big user space programs. And there's nothing else than those four main components. So if we dive a little bit more, and I'm not sure how far I am in the talk, whoa, time is flying. Another interesting part to discuss when talking about toolchain is the concept of C-sroot. So remember when I mentioned the build process for binutils, there was this dash dash with C-sroot configure argument. So what is this C-sroot thing? And this C-sroot is the logical route directory for headers and libraries. That's kind of the official definition from, I think, the GCC documentation, or whatever, or maybe binutils. So this is where GCC automatically looks for headers. So when you're doing sharp include stdio.h, where is GCC looking into in what we call the C-sroot? And this is where LD looks for libraries. So when you do dash, small l, I don't know, foo, LD is going to look for leap foo in the C-sroot. So that's where we wanna put all the headers and libraries that are going to be used by your compiler. Both GCC and binutils, when you create a C-sroot aware toolchain are built with dash dash C-sroot equal some location, which obviously is the same for the two. And what we do is that we install the canada headers, the C-library, and its headers all in the C-sroot so that your compiler automatically finds them at the right location. This C-sroot is an absolute pass on your build machine. So the question is what happens if you move your toolchain around like you move it into a separate machine or a different location in your system? Well, GCC is an LD or a can of smart enough that if the C-sroot is a subdirectory of the prefix, then it will automatically be capable of recalculating the relative location of the toolchain and its C-sroot and be capable of finding the headers and libraries even if the toolchain was moved around. However, if the prefix and the C-sroot are completely separate in different places, it won't be able to find them again, right? So most of the people who build toolchains that need to be relocatable, that you can move around in your system or install in other machine in other locations, have the C-sroot as a subdirectory of the prefix. You can override it at runtime using the dash dash C-sroot GCC options. So for example, in build-root we leverage it because we create in our own C-sroot where we install all the libraries that you build for your embedded Linux system. So we override the C-sroot that's used at using this runtime option. And you can at any time print the current C-sroot using the dash print dash C-sroot GCC option. Most toolchains have just one C-sroot. So they have one copy of the C-library, one copy of the candle headers, one copy of the C-library headers. And that's just fine. But those libraries, so the runtime libraries for GCC and the C-library, they are built for the target, right? So those are meant to be installed on your targets. They are contained ARM code or MIPS code or Pro PC code or whatever CPU architecture or targeting. So at the time you build your toolchain, you are deciding what optimization level and what exact CPU architecture variant you're targeting. Are you targeting RV6, RV7, RV5, with swatting point, without swatting point, with this ABI or that ABI? You're deciding that at the time you build a toolchain because the toolchain is not just the cross-compiler, but it's also a set of runtime libraries that are built for your target. So if you have a toolchain that is like contains, I don't know, an RV7 builds C-library and your target is RV5, no luck. You will put the C-library on your target and it's not gonna work. So what you can do is of course build different toolchains for the different cases you have, but that was not to the taste of some toolchain providers. So they came up with a mechanism called the multi-lip toolchains. So the idea of multi-lip toolchains is that you have a single compiler, but you have multiple CIS routes. And the different CIS routes have been built with different CPU optimization flags that target different platforms. So let's take a very specific example. Mental graphics is providing some, a number of toolchains, infinitely less than they did in the past, but one of them is for ARM and it's a multi-lip capable toolchain that provides three CIS routes. So if you do GCC dash print dash multi-lib, it gives you a list of three entries. So there's the dot entry in the default one, ARM V14 and sum two. And then next to that, you see the some GCC flag, so M arch, ARM V14, M thumb, M arch, ARM V7A. So these toolchains provide three CIS routes, one for ARM V4F, which is the default one, one for ARM V4 and one for ARM V7 thumb two. And the compiler is going to automatically use one or the other CIS route depending on the GCC compiler flags that you have passed. So if you just say GCC ARM V5 TE print CIS route, it's gonna show you the main CIS route which is in this weird location. If you pass ARM V4 TE print CIS route, it gives you a different CIS route location in which there is a different variant of the CIS library that has been built for ARM V4. And then if you pass other GCC arguments like ARM V7A, M thumb, it's gonna give you a different CIS route. And if we look at the different libraries in the CIS route, we have here, I just dumped with 3DLF, the architecture for which the dynamic linker has been built, so LDSO is the dynamic linker. We can see that each of them has been built for a different CPU architecture. It's the same, it's ARM, but it's different variants of the CPU architecture. So that's what multi-lip tool chains are. Just multiple CIS routes built inside the same tool chain to provide a little bit more flexibility. To continue this talk, what I wanted to do is kind of look at what's inside a tool chain. So I took a built route, built tool chain, did find slash, find slash find dot and look at all the files that we have in there and try to come up with some reasonable explanation on what all those files are. It was not that easy for all of them, but I tried this exercise. So at the top level of a cross-compilation tool chain generated by built route, you have something like that. So you have one directory named after the tool chain tupper, and then you have the usual bin including other directories. So if we look at the first directory that's named after the tupper, we have a first sub directory called bin in which we have the small sets of binutils programs that have been installed in there without their cross-compilation prefix. So they are just named AS, LD, NM, and read DLI and so on. So it's not a full set, but just a subset of them. And those are actually our links to the actual binutils programs which are in the bin folder that we will see later. In the bin folder, they have a prefix like in that case ARM, built route, Linux, Usylipsik, new ABI, HF, AS for the assembler, but they don't have such prefixes in this bin directory. And this is where GCC finds them. So as I said earlier, GCC is a compiler driver. So it's orchestrating the build. And when it needs to call LD, when it needs to call AS, it's not calling like your ARM build blah blah blah AS or blah blah blah LD. It's actually calling those binutils without the prefix. Another thing that you find in there is the set of headers for the C++ tenor library that were installed by GCC. For some reason, they are not part of the CIS route itself. I'm not sure why it puts them elsewhere, but this folder is anyway part of the default header search pass for GCC. So it finds them naturally. Then in lib, that's the location where GCC installs its runtime libraries. So they are built from the target. So we have LibAtomic, which provides a number of software implementation for atomic operations when they cannot be implemented directly by using CPU instructions. LibGCC, which is the main GCC runtime, it has a bunch of optimized functions. It has also a 64-bit division for 32-bits architecture, floating point emulation, and a bunch of other things that GCC doesn't provide on its own, but are needed on the targets to be able to run your programs. Transaction memory library, it has the standard C++ library and a subset of LibSTD C++ with only the language support, but if you really do C++ application, it's LibSTD C++ that will be used. And then we have the CIS route itself, which is really what I was referring to earlier. And this is where the C library gets installed and the kernel headers and the C library headers and so on. Then we have the bin folder that's the one most people look into first, because that's where you find your cross-compilation programs, Binutils and GCC. They're also wrappers for LTO, but as I said earlier, I'm not going to cover LTO in this talk. In the include folders, that's where we find the headers of the host libraries, but they are in fact not needed at all to run the tool chain. It's just an artifact of the... We've built those libraries so they've installed their headers, but we know that GCC has been built. They are kind of not really needed anymore. In Lib, we find, so last point, the host version of GMP, MPFR and MPC, which are needed by GCC, built for the host machine. And then we also find a bunch of other things installed by GCC. So the CRT begin, CRT end files, which end all the constrictors and destrictors, and then those are linked into your executable. So that's again one of the work that GCC as a compiler driver does, is it goes linker with a bunch of arguments to make sure that your program will work and it gets linked with all these small object files that are needed to make it work on the target. It provides a number of headers provided by the compiler. So in fact, you not only have headers provided by the canal and by the C-Library, but also a few headers provided by the compiler itself. And then there's a bunch of other things ready to fix includes that's a GCC process to fix up header files that come from the system. It's kind of weird stuff. And in there it also puts the static variants of the GCC runtime libraries. Another thing that GCC installs are LD scripts. So depending on your architecture, it has a huge bunch of linker scripts which are used to link applications or link shell libraries. And then depending on the flags that you pass to GCC, the different set of linker scripts might be used. So it just tells the linker how to lay out the final binary that will be produced. How to combine the sections together and things like that. If we move on to the next part in LibExec, that's where we have the actual compiler. That's where we have CC1, CC1 plus, so the actual compilers. And then there's a program called collect2 which is what GCC calls to actually do the linking. So it's another wrapper around LD. So it's doing some stuff and then calling LD. And there's a bunch of other things related to OTO and I want to get into the details here. And finally, documentation, translation file that's a little bit less interesting. Since time is flying, I'm gonna speed up a little bit. So architecture tuning. So GCC provides a number of configured time options to tune for a specific arch or CPU variant. So with arch, with CPU, with ABI, with FPU. So what they do is that they define the default value for M arch, M CPU, M ABI, M FPU and so on. So if you specify those at configured time, then when you call GCC, it will by default produce binaries or libraries optimized for this specific CPU variant. They can always be overridden at runtime using the corresponding M options. But again, as I said earlier, be aware that parts of the tool chain have already been built for the target. The C library and the GCC runtime libraries, they have already been built as part of the tool chain build process. So you need to make sure that they are, they have been built with CPU architecture tuning that matches your actual target. Yeah, one thing that's interesting is look at the GCC documentation. They have a very long machine dependent options where you can see the, on a per architecture basis, what are the accepted options and their values. So for ARM, you can see for M arch, what are all the possible values for the different CPU variants that GCC supports. Another thing that's important to know when discussing tool chains is the story of ABI. So ABI from the point of view of a tool chain, it defines the calling convention. So calling convention is essentially how function calls are made, how you pass arguments, how the, if they are pressed on the stack to registers, how the return value gets returned, the size of basic data types, alignment of members and structures, and when there's an operating system, how system calls are made. And from a, for a given CPU architecture, it provides registers and provides CPU instructions. They are potentially an infinite number of ABI's that you can imagine. And so it's not necessarily that one CPU architecture has one ABI. You may have multiple ABI's that have been invented or created for that CPU architecture. And if you have two object files that have been bid for different ABI's, you cannot link them together. Because of course, if they disagree on what the calling conventions are, like one thing that to pass the register, you do it all on the stack. And the other one things that the register are, the arguments are passed in registers, obviously linking them together will not yield something that works very well. So when you link object files together, they need to use the same ABI. So here is an example on ARM, which is probably one of the architectures where this is, I guess, the most confusing. There's a history of three ABI's on the ARM Linux world. OAPI, which is now completely obsolete and is not really necessary to talk about it anymore. It's not even supported in GCC anymore and it's been removed for a long time. I still wanted to mention it to show that there is an history of ABI's. So nowadays we have two ABI's in use on ARM, EAPI and EAPI HF. EAPI is an ABI that allows to mix hard float code, so code that uses floating point instructions with soft float code. So code that emulates the floating point operations so that it can work on processors that don't have any floating point hardware. And to achieve that, what it does is that it's passing the floating point arguments into integer registers. This way it doesn't assume that there is a floating point unit in your hardware, but it can still use it if there's one. But this passing of floating points arguments in integer register was causing kind of a performance slowdown on platforms where you really have always a floating point unit. So people came up with a kind of a variation of this ABI called EAPI HF, so the kind of the name suggests it's hard float, where those floating point arguments are passed in floating point registers. So it requires the use of an ARM CPU that has a floating point unit and its corresponding registers. And of course you can't link EAPI code with EAPI HF code together. So if you have a pre-built binary provided by your vendor, which is EAPI, then you can only use in EAPI applications with EAPI labories. Another thing I wanted to mention is difference between tool chain and SDK. Sometimes people are confused by that. Tool chain is a strict stance of it. It's just the four components that I mentioned, the compiler, binitils, C-library, and the candle headers. And that's just it. And a number of tools, especially build systems, they allow you to create SDKs. So SDKs are a tool chain or commented with a number of libraries that match libraries you have available on your target. So it's a compiler, a C-library, binitils, plus maybe networking libraries, crypto-library, graphics libraries, and so on, and the headers that are available on your target. And such SDK allow you to build a user space application that link against those libraries and then of course allow you to run them on your target. So an SDK is really more than a tool chain. So how can you get a cross compilation tool chain? You can then get them pre-built from your distro. So Ubuntu, Debian, and most of the Linux distros these days have pre-built tool chains for various architectures. There are also organizations that provide tool chains. So Linaro provides ARM and AR64 tool chain, Mentor Graphics has a few, but as I said, less than they did in the past. Imagination, which owns the MIPS architecture, provides MIPS tool chains. There are other vendors that can provide them. But you can also build it yourself. You could do it manually, as we've seen. It's not that complicated, but there are still a few gotchas here and there. But there are some interesting tools that will help you doing that. One of them is Cross Tool NG, which is really specialized in building a cross compilation tool chain. So there's a mini config-like interface which allows you to say which C-library you wanna use with GCC version, which can header version, which binches version, what is the CPU architecture variant you wanna tune for and other things like that. And then you can just say go, and it builds the whole thing for you and produces a nice tool chain that is relocatable and that you can share with other people. So it's a very nice tool. It's probably the most configurable and versatile option in terms of building tool chains. It can produce tool chains that target window systems and really other crazy situations. But also build systems, building embedded Linux systems, usually know how to do that as well. So if you take Yocto or Open Embedded, Buildroot, Openwrt, and many use other tools, they know by themselves how to build a tool chain as part of the overall build process for a Linux system. And pretty often they can also reuse existing tool chains. So if your vendor provides a tool chain pre-builds, you can usually use that as an input for those build systems. A few references, Cross Tool NG has a really great documentation on how a tool chain is constricted. So even if you don't intend to use Cross Tool NG, I recommend looking at that documentation for information. And the GCC and business sales documentation are actually interesting. It's quite maybe crazy to say it, but it really contains a lot of details on the GCC runtime, on the different options that you can use, and it gives lots of insights on how all the tool chains are working. Any questions? Yes, please, listen. Only that a build failed because applications that use GLib want version symbols in the standard library in a LubeC. Using GLib or GLibC? GLib applications, like GTK, want version symbols in LubeC. Yes. Which I've not seen anywhere else. And so our tool chain had unversioned symbols in GLibC. That seems kind of weird, but then, so as I said, GLibC provides backward compatibility for its API so that you can replace GLibC without recompiling your programs. And one of the way it does that is by using version symbols, so it uses symbols with ads, and then the version of the C library. You can disable that when you build GCC. Yeah. You can build a GCC without versioning. And then in that case, it might break existing applications that rely on those version symbols. So it's probably a tool chain that has been built without GLibC symbol versioning. Of fucking with people. Yeah. No, that's not what I'm saying. That's what it's called, right? Yeah. There's no reason, I think. I except for a specific embedded system where you know... Yeah, if you don't care about backward compatibility. But if you need the size of the GCC, you might need something like that as well. Or if you are security conscious, you want to avoid all symbols from being called in a rough attack point. Yeah. Please. Can you explain what's the Canadian build? Yeah, so the Canadian build is what you need when you have a build machine that is different from the host machine that is different from the target machine, right? So I only covered the case where build and the host are the same, so it's like you're building on your laptop a tool chain that you want to run on your laptop to produce code for different CPU architecture, different operating system. The Canadian build is what you will need if you want to build on your laptop, a tool chain that will, for example, run on windows and generate binaries for Armournex. That's a Canadian build that you need. And then more steps are needed to do that because of course, to allow your laptop to build some binaries that run on windows to build the compiler itself, you already need a cross compiler, right? Because you need to allow your laptop to build binaries for Windows. So you first need to build a cross compiler, which you use to build the compiler that will run on that Windows machine and produce binaries for Armournex. That make sense? No. Yes, you still need to build lips on Arm, so you also need, so you need one cross compiler to build the compiler itself and you need one cross compiler to build the Lipsy for Arm. And then you can go ahead and build the process. You need three tool chain builds. I'm not sure why Yocto does it so complicated. I'm not sure that's the reason. I think the reason why Yocto might be more complicated than the process that I mentioned is that I don't know if they're doing it, but they might be bootstrapping GCC, which is different from doing a Canadian build. Is that the problem with the process I highlighted here is that I'm building GCC with the GCC that is on your machine. So if you have a different GCC than the one I have, then the tool chain that we produce is different. So you can argue that the binaries that do tool chain will produce may be affected by how the compiler itself was built. So if you want to avoid that, you need to do a bootstrapping process where you first build with your compiler, and using that compiler, which is the same for you and me, we build the cross compiler. So maybe they're doing that, but I'm not sure. I've never been to the Yocto tool chain logic to build tool chain. For us, it takes about 15, 20 minutes to build a tool chain, something like that. Even less on high speed, high fast machines. Yes? So as I mentioned, if you build an arm tool chain, the compiler itself is not specific to a given architecture variant. So your compiler will be able to produce binaries for whichever arm CPU variant you want. So you can say, please build for context A8, build this for context A9, A15, arm 926, and so on and so forth, no problem. That's the compiler part of it. But as part of the tool chain, a number of target libraries have been built. The GCC runtime, the C library, the dynamic linker, and all sorts of things. And so you have to verify that those have been built as part of the tool chain build process with the CPU variant that matches your target platform or at least is compatible with your target platform. Does that make sense? So yes, the compiler can target any CPU variant, but the libraries that are built into the tool chain as well have been built for a specific variant that you need to make sure matches your platform. Yeah, so if I see a tool chain which is, I know it's specific for arm, but if I just see arm, yes, I cannot be sure which the libraries are. So you can be, not just by looking at it, but you can do, what was that, multi-lip stuff. Yeah. Yeah, if you do GCC-V, it's gonna show you the ways how it was configured so you can see the dash-dash-wiz arch, dash-dash-wiz CPU and so on, which it was used to build GCC, and most likely those were the tuning that was used to build the C library, but you want to be really sure you can use really a left dash A, and it's gonna tell you what is the architecture variant for which the C library was produced. So it gives you a very sure information that's okay, it has been built for MV5 or MV4 or MV7 with such or such optimization level. Right? Well, thanks a lot for attending. I'll be around if you have more questions. Thank you.