 Let's go. So, hi, everybody. Before I introduce myself, just a few words, I'm going to speak about Fedora. So, Fedora is a Linux distribution based on RPM, and it has several releases. And the one I've been using during this talk is Rohide, which basically means the latest version of each package is going to Rohide, and then they settle down. So, it's basically the same as unstable in Debian. Okay? So, we are using that as a testbed. All the binary package in Fedora are compiled with GCC, and that's a strong requirement. And basically, we're trying to switch to Clang, not for the distribution, but just to test both the packages and Clang so that we can gather more information. So, that's the overall topic of the talk. So, I'm Serge, I'm working for Red Hat as a compiler engineer, mostly on LLVM. And we're going to speak about Clang and Fedora. But before we speak about Fedora, let's speak about Firefox. So, this Fesco Steam stuff is a Fedora engineering stream committee, and whenever you want to do something that does not obey the rule of Fedora, you need to open an issue, and then there is a discussion, and something happens or doesn't happen. And this particular discussion was interesting because Firefox upstream decided not to support GCC anymore as their native compiler, which means that for both OSX, Windows, and Linux, the package is compiled with Clang. There are two main advantages of doing that, which are, if you take apart the quality of Clang, of course, is you need to support one compiler and not one compiler for Windows, one compiler for OSX, and one compiler for Linux. You've got the same compiler, so whatever compiler-specific stuff you're going to support is only going to happen once. So, that's cool for them. And the other more technical thing is that Firefox is being partially rewritten in Rust, and the Rust compiler is using LLVM as an internal representation, and Firefox is still using C++. So, if both use the same internal representation, it opens a lot of opportunities for optimization at link time, which is not possible if the C++ code is compiled with GCC and the other part is compiled with LLVM. So, there are good motivations for Firefox to switch to Clang, but then they still accept GCC fixes. That is, if the Firefox code base does not compile with GCC and you've got a patch, they accept it, but they don't test it anymore. So, as a packageer, what should you do? Either support the GCC build, which is a lot of effort, or switch to the same compiler as upstream and use Clang, which is okay because Clang is also packaged in Fedora, but it does not obey the rule of Fedora. So, they ask for an exception and first they got it and then the exception got removed, and that's the topic of that thread. Why did it got removed? There was quite a lot of objection to using Clang and without going into the details, some of them were related to the quality of the debug information generated by Clang compared to the debug information generated by GCC, which is a relatively well-known topic. Another one was related to the default flag like, for exception only, whether asynchronous unwind was activated by default or not for all targets. And GCC and Clang don't have the same behavior to that respect. A more interesting one was the support of semantic interposition. So, semantic interposition is an L thing that states basically that any symbol that is exported in a shell library may have its behavior or its implementation change at runtime, for instance, through a LD preload, which means that it's a strong optimization barrier for any interprocedural optimization. And by default, GCC obeys the ELF rules and Clang doesn't. Clang did not have any flag to enforce that rule until very recently. And so that's a big difference. It's a big difference for performance. I know that in Fedora, the Python library switched to no semantic interposition for the GCC build because of the performance impact. But by default, it's not the case in GCC and Clang has the opposite behavior. So that could be a blocker. A very important topic was security flags. All Fedora packages are compiled by default with a set of optimization flags and you've got, of course, minus O2 plus basically most of GCC security features turned on. One of them is 45 source. It's more for C code. And it was not well respected by Clang. If you activate that, Clang compiles your code correctly, but the security ordering does not actually happen on all situations, which is a problem. When I'm saying Clang, that's Clang 9, not Clang 10. Someone claims that the stack protection thing were not as evolved in Clang as in GCC. They rightfully pointed out that stack clash protection was not implemented in Clang at all while it is in GCC. So basically, one of the strong opposition was if you compile your code with Clang, it's less secure with some definition of secure than if you compile it with GCC. And that led to Firefox still being supported for GCC by the Fedora guys, which is for, from a maintainer point of view, that's not a very good property, but that's how it is. So we told ourselves that's a good opportunity to improve Clang and maybe to remove some of this constraint. So the Fortify, so I had a look to that. And for Fortify, it was very interesting. The way Fortification is implemented in the GCC is basically you've got your memcopy and then if Fortify is activated, you've got an inline definition that overrides the behavior of memcopy. But the Clang behavior was if you use a built-in and memcopy is considered as a recognized built-in, then you don't care about any overloads that would be given by the user. You're using the actual built-in semantic. And so they were not using the Fortify semantic because it was recognized as a built-in. So we changed that. And now when you compile with Clang and Fortify on and you found a memcopy, if there is an inline definition that changes the behavior of memcopy, Clang is going to honor that version. And so you've got the runtime checks and so on. So that's one more step forward. That's not for Fortify specification. Well, there is no Fortify specification. There is just the GCC implementation which is tightly linked to the GCC one. But if you want to have the same observable behavior, you're getting closer. So that's one good thing. There's also warnings. At some point you could say, and many packages do compile with W error, that if GCC detects a security issue and warns about it, maybe Clang should warn about it too. And one of the issues was Sprintf. If you can compute statically the size of your buffer, then you can do some kind of checks whether the format is going to fit into the buffer or not. And GCC has some not that much advanced analysis on that topic. And Clang has a few of them to a lesser extent. And now Clang implements full understanding of the printf specification and can compute the lower bound of the buffer size requirements and warn you if the buffer is not large enough. So I don't expect that warning to be issued a lot of time, but still that's something that can be now detected. And it has its end in Clang, but probably not Clang 10, it would be in Clang 11. And that's something that GCC does not implement now as much. So we are raising each other, but that's the same emulation, so I'm totally fine with that. About semantic interpolation, that sounds like a very difficult thing to implement because it basically means that whenever you've got an inter-procedural optimization in Clang, you need to change the behavior of this optimization depending on the status of the symbol based on basically its visibility and linkage type. The good thing was it was already implemented at the LLVM level, but there was no switch to activate that in Clang. So what looked like a tremendous task actually was not that difficult and which made it possible to land it in one week or so. So now the default in Clang 11 is still no semantic interposition, but you can force the flag to minus F, semantic interpositions and it will be respected just like in GCC for a package where it matters to have symbols change at both load time or run time. Just to give you an hint about the impact, if you've got this small code with semantic interposition on, you can inline full, but you can't inline bar because bar could be changed at run time and so changing its implementation would not have the same impact. If you inline it, then the LDPR load thing will not change all the behavior of bar. Then there is stack-clash protection. Funnily enough, the stack-clash vulnerability is one year and a half old and the flag got into GCC relatively quickly and not at all in Clang. The basic idea is whenever you allocate memory on the stack, if it's larger than one page, then there is a risk that it overlaps the page words allocated by the kernel and so your stack is going to overlap with your heap, which is not the property you want and that's the beginning of the attack and so the contour measure is basically don't allocate large stack but allocate by chunk of one page and touch memory each time you do that so you're sure that you will trigger the page words allocated by the kernel and then you've got your signal and you're safe. So there is an implementation for x86 under review. It turned out to be a bit more difficult than expected and it's only for x86. It's obviously very architecture dependent but you can see on that small example that you are allocating a large amount of stack and instead of having a single sub, you have two of them with a move interleaved which actually touches the page. The good thing with x86 is that whenever you make a function call, you're saving the return address on the stack so you've got a lot of free probes in the stack so you can implement that locally at the function level and it may not be the case of all architecture which is why it's not implemented at a higher level. So with that set of patches, most of the points raised during the Firefox stuff got handled and so maybe the situation can evolve but it also rings a bell. What about the next package I want to switch to Clang upstream and what are we going to do? Or maybe we could state it otherwise. We learned a lot by using Clang as a compiler for Firefox within Fedora. Maybe if we don't compile only Firefox but all the package, we can learn much more things. So that's what we did. Basically changing the default compiler in a separate build route in another Fedora build and gathering information and that's the second part of the talk sharing some of the insight we get. So we're still at the information gathering step so no answers but good hints about where we could drive Clang and that's for a real package. So all the data collection, the setup of the build route and so on was done by Stomstela so huge thanks for him because I'm not proficient enough on that part and so he did all the dirty stuff and I only had to grab full logs which is totally in my scope and then starting to implement the thing in LVM which I could start to work on. So it's not about Clang but it's about Fedora. Fedora-Rohide requires Python 3. We could still depend on Python 2 but well the word is moving so let's move with it. Well the word started to move 10 years ago so it's a good time to start to move on our own. Many tools and I'm not speaking about utility scripts but tools that the end user actually sees depend on Python. So that's especially the case for LLDB for all the debugger scripting parts but that's also the case for Clang Format or Clang Format Dave. And so if we want to ship these scripts in Fedora we need to have these scripts run correctly in Python 3 but if you have a look at the requirement for LVM it's Python 2.7 still now. So we had to port all the scripts to being portable between Python 2 and Python 3 which was quite a huge task, not a very difficult one because we are not using a lot of string manipulation in the script so that was basically okay but still it was some kind of an effort but now all the Python code base of LLVM is compatible with both Python version and I start in a thread which received relatively good feedback that by the end of this year we could probably drop Python 2 and switch to Python 3 which would not be and Python 2 is not supported for this whole year so it totally makes sense and LLVM doesn't have a strong history of supporting aging technologies so we are upgrading the same version quite often so we can also upgrade the Python version but at least that was the requirement for Fedora and this requirement is fulfilled. So another specific tool to Fedora is Anubin so Anubin basically it's a plugin for GCC and it records all the compiler flags into nodes so that for each function in the binary we know which compiler flags were used and that's mostly used for security you know which security flags were used for each function and then you know for your whole binary if some functions were not compared with the appropriate flags so that's a GCC plugin and it relies on some specific linker supports basically it was about nodes where attached to function for elf groups and when you merge groups the nodes got removed by LLVM and it's no longer the case but it was something we needed to implement and if we want to make one step further this plugin, this Anubin plugin could also be implemented for Clang and that's being done internally at Redat by a colleague of mine with my support so as this plugin is used by default it was important to do that task then about the real normal package now that we have set up a decent environment how do you change the underlying compiler because we've got all these spec files the spec file is just a recipe about how you build your actual Fedora package how do you change from the system compiler to Clang it turns out that the easiest way to do that was to provide a new set of RPM macro basically stating that whenever you mention the person cc is going to be redirected to Clang and the same thing for make, cmake and so on it assumes that everybody is writing an abstract description of the build process which was not the case at all but fixing that is also a way to fix package description so that was a good road to follow and once we get at that we get some builds that fail we get logs and we can grab for that to understand why we've got errors so that's the basic process and that's the result so on the 4000 package we rebuilt three-quarter of them just rebuilt without any issue which is both a good surprise but it also means that 1000 of them remains some of them didn't build with GCC either I mean because of some reason which are out of my scope so we put this one apart and so we are interested in the 1141 one that actually failed and so I had to dig through the error logs and understand why and so almost 300 of them just failed because they were compiling with Clang but using some GCC specific flags and so we get an error so we also had some Clang specific flags that were added by us in the process and they were not supported by GCC which means that our instrumentation of GCC failed which is also a good information to some extent so StarCache protection is not respected but we knew that so some arm-specific things about neon and quadrup accuracy floats something about how you pass arguments which we could probably ignore in Clang there's already a bunch of flags that we just ignore so we could add that one the tracking is to improve the quality of the debug information so we can relate that to the quality of debug information in Clang but we could probably ignore that one too because it's just debug information it doesn't affect the user experience at all no Trample line is more related to GCC implementing nested function through Trample line so it's a way to control that and no enforce exception only expects it's something to lighten the cost of exceptions or enshrink the binary size so maybe we could implement that I don't know sometime we just failed to hook Clang instead of GCC and we detected that so to detect that we just go through all the binaries generated well shipped in the package and we go into the command section of the binary and if we don't find a compile with Clang then it means that we did not compile any objects if we file compile with Clang it means that at least one object of the package got compiled with Clang which is not all of them there is always unfortunately the GCC well unfortunately for our tracking thing the GCC command because we are still using the CRT stuff and GCC as a linker so there is always the GCC watermark but at least some of them were not compiled at all so we need to understand why some people find it smart to compile with W error and for our task it's not smart at all because Clang and GCC differ a lot to the extent of the warning implemented and what kind of situations they catch and so something like printf being redefined is not an error on GCC and it is well it's not a warning on GCC but it is in Clang inuring virtual value detective infinite recursions this kind of thing so it's probably we are not going to implement the same set of warnings so I don't quite know what to do about that one probably remove W error would be a decent approach but I'm not going to spend much time on that some when you activate fastmat you've got some specific implementation of the mat library and it turns out that GCC relies on some new symbols that's finished by Finit and these symbols sometimes we fail to link with them when using Clang I can't say why but so it could either be a packaging issue or a Clang issue I don't know but still it means that there is some implicit things are coded in GCC or in the package that are not well described the famous configuration error we have got a lot of them you may be aware of the difficulty of writing portable auto-conf tests and if you're not having a look at all the tests that fail because Clang is not capable of compiling a C code that's the only output we get from the configure we could of course have a look to the config.log but so many package failed just to configure because using Clang during the configuration step just fails my hint about that is that because we use some specific compiler flags they are used during the configuration step and so if we've got any of these W error things or unsupported argument things we're going to detect that at configuration step and so we just fail to configure does it mean that we need to rewrite the configure step which means contacting each upstream package to update that or do something else I can tell but that's something which will not be very pleasant to fix and maybe not of much interest with respect to just Clang something that is important for Clang but only in Fedora sometimes Clang fails to find it some part of the runtime either because it's not well packaged or because there are some assumptions that are not met I can't... I am doing the packaging I do hope that it's not a packaging issue but I fear it is and we need to check that I can't tell about the GCC and the Scorsche and I did not took time to investigate what this implement but we can discuss about this a bit later sometimes there's disagreement on the standard so that's something that is very valuable either for Clang if they wrongly disagree or for the upstream package if it means that they could improve their code base so sometimes it's linked to GCC extension so the one like you don't support nested function sometimes it's comparison between pointers should be equal or different but not ordered comparison and I don't know if what the standard actually says about that but obviously GCC and Clang disagree so maybe we could improve the situation and sometimes there is just use of undeclared identifier and I will be surprised to investigate and check why Clang doesn't find identifier while GCC does so that's more of an upstream issue so the language extension stuff here you have a nested function something you can easily write with a lambda in C++ but GCC allows to do that in C for an extension and some package rely on that and it's probably going to be difficult to make them change their mind because if they choose that extension it's because it makes their development flow easier or I don't know and when you get you try to compile that code there is a warning like OK Clang is not supporting that Clang is not going to support that so it's clearly stating in the error message that we can't do anything about that we could maintain a patch at the federal level but I don't want to do that OpenMP and it's also related it's probably related to either a packaging issue or dependency issues sometimes Clang doesn't find so the runtime support for OpenMP or the configuration step fails and fails to detect OpenMP when compiled with Clang we need to investigate because that's not a good thing if Clang users can't use OpenMP out of the box in Fedora that's probably not a Clang issue more probably either a configuration issue or a packaging issue and some package just still use GCC so there are art codes make or the art codes GCC or G++ either in the package or in the actual upstream sources which makes sense because if you're not sensible to portability you just use GCC and it works and there's no reason to worry about that until you realize that some users may want to compile with Clang with good argument or Clang or whatever it could be Intel compiler because of performance I don't know and then it could be an improvement both for the packaging and for upstream to have a more generic build process so that's the end of my talk the ideas are a good life to share I'd like to share with that talk was there's a lot of learn there's a lot of thing to learn while having such a diverse set of program to compile with Clang and that's far from the test suite of LLVM so a good opportunity to improve probably making all this information available in a way that anybody could access them opening all the bugs on Bexia or on GitHub depending on how it evolves could be a good thing it's probably less important for the LLVM community to do the upstream work I mean trying to update every package packaged in Fedora to support Clang is more of a social task than of an engineering task so it requires special skills but some people may be open to that and some may not but that's still an opportunity for upstream to improve too I see a lot of good things for Clang at least for the past months when I've been working on that and so it also fills my daily job on interesting tasks so on a personal point that's a good thing and I wanted to last with that final error so if you're compiling in C++ and you name one of your member function dprintf and then you compile with 45 source but you order your includes in a way that sometimes you will get the macro definition from the STDIO headers and sometimes you don't then you end up with part of your class having a dprintf member and the actual implementation using another name and so you've got an error just because of basically macros and the way they interact with 45 source and that was a funny one only one in the world of Fedora code base but an unexpected one so now if you have any questions I'm happy to share more information Thanks for your talk I was curious about you mentioned earlier a GCC plugin about recording the exact compiler invocation in the notes section of the ELF object that sounds generically useful theoretically that could be a dash f flag in the compiler it doesn't need to be a plugin, right? So the question was recording all the optimization flag and security flags is something that is generally useful and so while using a plugin and not an actual compiler flag maybe I'm not the one developing that plugin the plugin is open so that's not a property reason but that could be my guess would be it's easier to write a plugin it's maybe the first step before you go into the code base that's how academics do they don't write into the code ELF code base they first write plugins and eventually it gets in so a plugin is a good first step so that could be an engineering answer would be why not starting with a plugin it's easier you can maintain it you don't have to get through the review process of the actual compiler and should it evolve into an actual compiler flag it does make sense to me at least yeah, whatever order Thank you for your interest in talking and many interest in details Let me wonder do you consider this journey a success for you and also like as you still got a lot accomplished what are the most valuable outcomes which you got for your day-to-day work out of this project for testing but for testing Fedora what kind of testing could you elaborate you'll get out of this project I don't quite understand the question can you repeat please you did this project for testing can you give more details what you tried to do so the question is we did that for testing what's next and what are the conclusion or the outcomes and so my personal take on that is Clang LVM did improve a lot thanks to this testing only on the Firefox part and so from all the information we gathered on the larger scale test there's a lot of situation where we could improve Clang based on the GCC experience basically and that's a good thing I mean it's not only about trying to match GCC so that's the initial motivation but in the end what you get is more feature for the Clang users and I'm going to go on that way because not for all the flags I'm going to sort out the things that I'm interested in and the things that seems valuable for Clang as a whole could be accepted because like nested function I could find it interesting on my own but it won't get accepted upstream so I won't work on that and it also provides opportunities to be able to work on that apart from just the LVM team in Red Hat so basically it provides more opportunity to fix bugs or add new features and driven by actual test case and not only my imagination Victor? Project to rebuild Debian using Clang Are you collaborating with this project? I'm collaborating with the Debian project on that so the question was there's not only Fedora the BSD systems some of them are compiling with Clang there is a side project of Debian of compiling Debian with Clang which has much more history than we have are we in our bubble or collaborating so it happens that I personally know the guy in charge of that for Debian and so we do collaborate both on the packaging side I have a patch that I need to apply to LVM I'm discussing with him are you applying that patch to Debian 2 why for this kind of thing and also for the general approach especially for Firefox because this guy appears to also be in the Firefox team so for instance all the Fortify stuff and the Stack Clash Protection stuff while being reviewed by LVM we are also tested on the Firefox version recompiled because they plan to use it they are really interested in that feature so even if they don't make it to the Fedora build they will make it to the Firefox build so there is some kind of cooperation one more question are you just testing that it builds no it pains me that you have so much so less trust into the package job whenever you do a package you've got a test section and basically you're running the test how many packages actually have tests I can only speak about the package I maintain in Fedora which is very small amounts but all of them do have a test section so it's also true that for some architecture the test section of LLVM has an if, arm, then okay but well we are actually running the test and ignoring the results but not for x86 but there is a test section and we are encouraged to use it and I think some I've not mentioned that but some of them we had sick faults in the test like the tests were run sick faults and from my experience working on obfuscating compiler we don't like debugging this situation because you don't know the tests you don't know the actual program and that's a pain but that's one thing that could be very interesting because if kind of is miscompiling something yes