 All right, let me wake somebody up So everyone's having a good morning so far and presenting LLVM Linux to Linux kernel with drag wings I present being Webster from Converse Incorporated. Please welcome being Yeah, so today I'm going to be talking about essentially Compiling a Linux kernel with Clang Alternate compiler to what is more traditionally used in the Linux community, which is of course GCC So the first thing I tend to get asked a lot and so I'm gonna go over it again Even though I know some of you already know the answer is what is Clang and or LLVM And that's the first thing people get confused about is is Essentially, what is it what is it all about and it's really two things the first thing is it's actually a toolkit It's a it's a set of libraries a set of pieces of code that can be used for building different things One of them happens to be a compiler You can build linkers their search code analysis tools There are tools for extracting metadata for things like documentation or what have you it doesn't really matter how you slice it Ultimately, you can use the same code as used for the compiler To parse and then the same grammar in order to pull out information from the the parse tree that comes out of that So things like code refactoring tools You can build into ID ease a lot of the stuff where we tend to have to re-implement the parser and the grammar in order to get That same goodness in other tools In more traditional tools with Clang and LLVM you can actually use the same code In your in your other tools that you want to build It's also a toolchain. So in fact there is a compiler the one we use is Clang It's a cc++ objective c compiler There's the equivalent of lib gcc called compiler RT again. That's a series of highly optimized Routines that are used at a low level. Oh, that's primarily used in user space not so much in that kernel space There's a couple of different linkers that are being worked on although. They're still a work in progress one of them is Similar to the GNU linker LDD Sorry, rather LLD rather the other one MC linker is designed more for linking on an embedded target. So it's not as Optimized, but it's fast and uses less memory okay, then there's a number of other tools like the static analyzer, which we'll talk a little bit about later and There's a debugger that again is essentially a drop-in replacement for GDP a Lot of other tools too. We're not going to necessarily talk about those today But that's in essence what LLVM is that a library is to build tools and a series of tools that build up a toolchain traditional toolchain Okay, so the next thing I always get asked and doesn't matter whether I'm at a airport or in the hallway people are always asked Well, why would you bother right? Why not just keep using the same toolchain? We've always been using it's been good enough for the last 30 years Why not continue down that road? There's nothing wrong with GCC. I love GCC. I've used it most of my professional career. It's great However, there are a number of things that alternate tool chains bring to the table There we go the first one that came to mind when I'm asked this question is And it looks like we've switched two slides fast compiles Okay, so one of the things about having a greenfield compiler is they were able to optimize everything down to make things as fast as they possibly can and in fact in a lot of the the Test that you can do for timing Ultimately you find that in fact in certain situations Clang actually outperforms GCC by a huge margin and in a lot of situations That's because they're sticking strictly to the standards and they're not going at some of the extra miles that that GCC is doing And there's a few features later that we'll talk about that Clang doesn't do because amongst other things it would slow things way Down because it complicates their architecture now I will fully admit that I have pick and chose chose the The slides here that make Clang look the best. Okay, I'm not trying to say that GCC isn't a good compiler It's just that there are situations where Clang outperforms GCC and a lot of other situations. They're very close Okay, so again, I'm not trying to put anybody down in this case Clang tends to be faster tends to use less memory Okay, the next thing is it's a really fast-moving project. I know a lot of projects in open source are fast moving Okay, certainly the kernel We see over and over again how fast it moves and anybody who hangs out on LKML can see how fast things move Rather dizzying for me one of the things about LLVM. That's really exciting is it's actually a pretty young compiler It's only been around for about Sorry LLVM has only been around for about ten years Clang has only been around for just over half of that and In fact in a very few years. It's actually caught up to what Clang sorry What GCC can do in a large degree? Right, it's faster. It's in many ways. It's it's as standards compliant or more standards compliant and Performance-wise, it's actually getting pretty close to what's possible again I've chosen a slide that happens to show Clang out performing GCC in this case both 4.7 and 4.8 Okay, not always the case. You'll find that when it comes to vectorization GCC still wins Okay, but again that's changing very rapidly you can see just from 3.2 to the 3.3 timeframe for Clang there's actually a market improvement that took us from The kind of performance that we're seeing in GCC to something that's considerably faster Okay, so this is another reason why you'd want to use it. Is it way way way better? No, but it is getting better at a very high rate okay, so We've caught up at this point It very well could actually surpass it in a relatively short period of time I'm not saying it will I'm saying it could all right The next thing is there's a lot of projects a lot of companies a lot of people working on different things all at the same time As soon as you start working with multiple processors multiple boards multiple products All of a sudden you have to start dealing with a whole bunch of different kinds of tools Now one of my favorite times is when I started working on projects where I could use GCC across the entire Problem space that made my life a whole lot easier Okay, one of the neat things about LVM is it actually pushes into a few places that GCC doesn't touch things like the GPU things like Vision on small microbes and that sort of thing and so you see it used in places like kernel space and user space libraries You see it used in applications certainly in mobile embedded It's very it's becoming very popular certain on certainly on the other non-android platform But you'll see it for developing Extracting documentation There's some code that does that much like doxin does or Java doc or what have you or kernel doc It's also very heavily used in HPC And the reason that is is because you can actually take the the interchange format What's called RTL in in GCC, but it's LL IR the interchange format that's being used and in In LLVM and they're using that to actually optimize what's happening in highly parallel cluster based systems All right, so you see it used in a lot of very targeted areas things like audio things like video replacing Parts of GPU you'll see in a lot of embedded systems now and systems with low-end video cards. They're using LLVM pipe That's what they're doing is they're implementing shaders and that sort of thing that are traditionally done in hardware in Software and they can do that through using the LLVM Technology You'll also see that CUDA right whether it's on AMD chip Like I say TI chipsets originally or Nvidia all the CUDA stuff There is ultimately based on LLVM and in fact render script on your Android phones You know those little animated backgrounds, you know smoke or Koi pond or whatever else that's actually running on the GPU Okay, and the reason the reason that can run on the GPU is because you're taking the render strip Which is a C-like language and compiling it down to whatever the GPU is on your actual phone at the time That's actually done on the phone using LLVM Right so it gets shipped as a as a interchange format, and then it's compiled out to GPU code on the fly on On the phone itself what that means is your CPU can go to sleep and your battery gets saved Right the GPU's got to be on anyways. It's driving the screen see if you can go to sleep Those are the kinds of things you can do with that and one of the reasons That having a single toolchain for this is useful is you start getting to the point where any Optimizations any extensions any things that you fix all of a sudden it applies to your entire problem space not just a part of it Because the problem places now got bigger All right, that's another advantage for using LLVM The next one is license Sadly, this matters to some people Okay, I'm a GPL guy and everything else, but what it comes down to is certain Situations don't allow you to link GPL code into whatever else it happens to be with BSD or proprietary Kind of situation the nice thing about LLVM is it's a very permissive license. It allows you to build it into your Your free project it also allows you to build it into a proprietary project What that means is again you can take that same parser that same grammar that same set of utilities and Libraries that are built around these technologies and build it into whatever you happen to be building now The side effect of the BSD of course is that people don't always play nice They don't always share their code back again what you see in the LLVM community is that a lot of stuff is sent back again Now there are certain things that aren't but in general The code is moving very very fast and people get that it makes sense to push code upstream And if you certainly hang out on on the IRC channel for LLVM you'll see people from a lot of different companies Pushing stuff up asking questions People get that pushing code upstream into LLVM makes sense Like I said fast-moving project Very helpful project to always people. They're trying to help The other thing that's really cool about this is you have a lot of full-time coders Extending and fixing and making the compiler better some of them are open source coders Some of them are not okay on The surface that doesn't necessarily sound like a good thing But that means that there are companies really big publicly traded companies that are paying people to make your compiler better Okay, that's not a bad thing Right the Google's the Apple's the other people the world that actually pay LLVM coders are actually contributing to making this this compiler better faster Smarter less memory whatever That's a good thing some of those people can sign NDAs which means they can sometimes push Certain optimizations into hardware even better Whereas open source coders can't always do that There are advantages to having that kind of thing It's a very wide development of our audience that develops it The other one that's kind of cool is something called static analyzer the client static analyzer and We just recently got this working We had a Google summer of code students and he'll actually be here later on if anyone's sticking around for the next conference He'll be at the the LVM microconference that we're having on Thursday He actually got the static analyzer working with the kernel which is really great Static analyzer is a little bit like all the other tools we have like coxnell and smash and and so on However, again, it's built on top of the compiler technology itself so instead of having to reimplement the parser the Grammar of C++ reimplement all the parts so they can understand the code It's actually using the compiler to take it through to the AST parse tree And then it's it's actually looking at that in order to figure out what's going on at the static analysis level This is a really really simple example But this is showing basically that if op-dark is null Essentially, we drop through and we get down to a point where we actually end up using it We actually pass null to string to ul Okay, now, this is really simple if you actually follow through some of the more Difficult or complex examples you'll actually find that it'll actually jump through multiple functions all over the place and come back again And 50 steps lead you from a set of inputs to a potential failure some sort of segmentation faults or or panic or what have you Okay, so there's a lot of really interesting neat things that can be done here now Most of the checkers that are built into static analyzer right now are targeted at user space All right, so user space applications and that sort of thing One of the cool things is that we have actually turned off a lot of the ones that don't we don't really care about like malloc and free for instance And we've turned around and we've started to add some kernel specific ones Okay, and the really cool thing is is that every once in a while in in the kernel community people find a common problem Or something that people are doing over and over again and ultimately people extend check patch or they add it to the coding standard Whatever else potentially static analyzer could actually be extended in order to look for these problems of very deep problems that cross functions and ultimately You know have deep deep problems that people can't necessarily find just by by reviewing the code So it's actually pretty interesting We actually have a We have actually have this running once a day on our build bot that we talked about later There's a link on our website that actually will show you what it currently looks like what the output of that currently looks like The only that Clang actually introduced that's very handy something called fix it hints Now fix it hints are something that I wish I had a long time ago Okay, and the cool thing is is that recently fix it hints to the same level that you can get in Clang have been added to GCC If you use GCC 4.8 4.7 wasn't bad 4.8 actually has fixed hints They're almost as good in in some cases are a little bit better than Clang Okay, again, they've caught up to each other. This is one of those situations We're having real competition between compilers is actually pushed the incumbent to actually come up to the same level Okay, cuz Clang really blew GCC away for for at least the last year and a half in this regard Okay, and the cool thing again, this is a relatively simple example and apparently the contrast is really terrible on this particular Projector, but it's showing you that there's a particular use of old GNU style feed field designators So instead of saying dot x equals it's saying x colon was the old way of Doing structure assignment All right, and so essentially what happens is it finds a problem. It tells you what the problem is and then it will actually Propose a fix And in fact one of our project members actually use this code in or with a little bit of Python magic to actually spit out new code Which was kind of interesting? Okay, one of the things that we I don't have on the slide anymore, but it'll actually do macro expansion to again Something that GCC now does so the neat thing is instead of saying there was an error on this line Which happens to be full of macros you go figure it out It'll say it's on this line here, which is in this macro, which is here Which is in this macro which is here and here's your actual problem Drilling all the way through all the macros and again GCC 4.8 now does that It didn't used to Now it does this is that competition. That's really making things better Another good reason for having to the next one is Actually a quote from Michael K. Johnson happens to be sitting in the near the front of the room here I didn't realize he was going to be here, but essentially on D plus the other day. It was actually Ted so's He's sitting next to him had a post the other day talking about about the random number generator and In the middle it was this this comment, which I really wanted to bring up and as essentially Michael was pointing out that having more than one compiler that have fundamentally different internals different Architects internal make it much more difficult for somebody to attack a system that is running a kernel With a compiler you don't necessarily know what it is. It's very hard to do like a remote timing attack or what have you on that Exactly reflections on trusting trust and of course there's a lot of other kinds of things too a lot of the the Kernigham and Richie Sorry the Kernigham backdoor attack that he did way back when where he he He trodden the login command and the compiler in order to always have a a backdoor into Unix systems That's something that that's not as easy easily done and certainly Approaching a possible when you start having more than one compiler and you can start comparing the outputs of those two compilers two independent compilers Okay, so there is an aspect of security that you gain by having a choice It's also lots of other really cool things. I encourage you to go and watch there's a talk that was given At one of the LVM conferences by following Google He built this really crazy tool that actually takes a look at the C++ code that Google has and apparently They've got a crazy amount of C++ code that runs all the stuff that we like using on their websites and he took a Tool built on LVM and he actually compiled it through to the AST parse tree and then look for common bugs Common API changes and so on and so forth a little bit like what static analyzer does but he compiled it through back into C++ code Okay, so it's going from C++ code through a series of filters back out to C++ code And if they're different essentially the original developer was sent an email saying here's the fix Does this make sense? So imagine LKML But the robots are posting the patches. It's seriously awesome. You got to see the movie. It's it's well worth watching My whole point is potentially something like that could be done for the kernel Okay, now this isn't directly related to using clang to build the kernel, but it's a tangential technology that can be used by Making the code actually work with this particular parser and grammar We get that potential side benefit as well Okay already used by a lot of different projects As I said previously it's part of render stripped It actually supports ARM MIPS x86 anyone who puts a handset out an Android handset out Ultimately has to make sure that renderscript works on their particular GPU rates with the handset problem You just have to write a renderscript program. It will run on all the different platforms as long as they support renderscript Again, it's used in a lot of video stuff for shaders and LVM pipe and so on and so forth and This is being used already in places like Ubuntu on lower-end machines that don't have good GPUs It's also something that is being pushed into projects like Yachto. It's something that When I was looking at on ARM and so on and so forth It's actually making it a lot into the embedded world where again our GPUs aren't necessarily as fast as they might as we might want them to be and Again, actually during the LVM microconference that we have later this week We actually have Sylvester LaDrew coming to talk about his work on compiling Debian with Clang And what he's been doing for several years is recompiling the whole Debian package archive with Clang and then publishing essentially how far he got and last I checked he was over 80% it sort of fluctuates and the reason this is because The code that he's compiling gets better The compiler gets better. Okay, the problem is unfortunately is they sort of get into parts where Clang finds more problems even though people have removed other problems and so you you get this bounce effect where Something it did work doesn't work anymore and something that didn't work now works and so essentially we're we're somewhere between 80 and 90 percent and It keeps on moving around but ultimately again What he's finding is is that there's a lot of code that is becoming more standards compliant and as it becomes more standards compliant and not Well, it doesn't use GCC extension so much ultimately Clang mostly just works Okay Which is a similar problem we have in the Linux kernel Now before we get to actual Linux kernel part itself Just talk a little bit about my projects the project I'm a part of and there's a few project members in the room right now essentially we are working to essentially port Linux to work with Clang and The idea that we really want to do is we want to bring together like-minded people There are a lot of different people that we're doing this put them all under a single umbrella get them to work together Ultimately to get the thing to work at all which it it mostly does right now Also discover problems that are happening as LLVM and the kernel progress New problems get introduced now. We're not there yet on the kernel side, but on the LLVM side We're already starting to find There are new features new bug fixes that are going into LLVM that then break other things So for instance, we have a target for at Nexus 7 it was working great And then all of a sudden it broke and the reason was is because effects went in for another problem that actually broke the Nexus 7 build and The guy that we have that works on that essentially worked with the LLVM guys and ultimately managed to fix the problem And it ended up being a problem that would have bit other people in the future But we were able to find it very rapidly So we found the bug within I think it was two days of the bug being introduced to their their SVN and Yeah, they use SVN You know instead of get but whatever But ultimately it was it was fixed within two weeks Which was kind of kind of nice and that was mostly because they couldn't get a hold of the developer in the first place Okay, so you know these things are getting fixed a lot faster than they used to And ultimately the project of course is to upstream upstream fixes to both the LLVM project and to the kernel community So we do this through a number of different tools We've got a an automated Build system originally we had to patch both LLVM and Clang independently as well as actually QMU and a few other things to get These things to work properly Basically having a great big document telling how to do this Really raised the barrier to entry so a series of scripts make scripts basically are put together to download patch The code build it so you can actually get it all worked together So it downloads the code for LLVM Clang it downloads any cross tool chains that are necessary We'll talk about why that is in a second the kernel code QMU and different test images And ultimately makes them all work together so that we can see whether or not there is a bug or not Okay, the whole build infrastructure is just scripts and patches. That's it Okay, we're not trying to fork anything We're not trying to you know get you to use our version of Clang or our version of the kernel No, we're trying to build on top of what's already out there. I like to call our project a meta project Okay, we're not trying to we're trying to add to everybody else's project not our own Okay, we want to make this work everybody else recently We added the ability to actually not only build from our version of Clang our patch version of Clang rather But to also use different sources so we can use the one that's built from source out of our scripts We can use a pre-built LLVM. We're actually at a point now where LLVM 3.3 now works out of the box with the Linux kernel so we can actually use the pre-built one from LLVM org There's packages there for whatever distribution you want to use or natively installed ones so depending on what district you're using you can just yum install app get install whatever the 3.3 version of Clang Now again, we'll talk in a second about why we need this But unfortunately still need the GNU assembler and the GNU linker To build the kernel. Okay, so in the cross situation We actually have to download one of those as well and make sure it's installed and again You have a choice for these are arm related, but code sorcery linaro android or the native built one Okay, again, if it's not there it'll download it'll install it for you You don't have to worry about that and you can choose which one you want to use of the command line so the targets that we we build and Try to test on as much as we can x86 64 obviously that's the big one We have to have to support first all expresses the one I do most of my work on and that's that's the arm side We use QMU for testing that Okay, we're actually trying to do the same thing with x86 64 We're not having as much luck there. Unfortunately, but I don't think QMU is not giving us the same kind of test coverage We're getting on the arm side. That's a work in progress. That's not Just we just we've been doing arm the arm tests for well over a year and a half the x86 stuff is only a couple months old It's just work in progress. That's all it is We're finding certain situations. We will boot under QMU and it works then we try it on real hardware and it doesn't essentially so We've also got a port to raspberry pi and the Nexus 7 and that's those are actually the two that have broken With LVM upstream recently that we've actually had fixes for I've also got some work getting the Galaxy S3 to work although that's had a couple of issues and My pet projects are actually getting the Beagle bone and arm 64 to work not that I have real arm 64 Hardware, but just to try and get that Get that compiling I'm most of the way there We try to automate as much as we can. We're not a huge project So we have a build bot that does this this code on a regular basis Downloads the code patches and so on We try to run an LTP test suite every night in order to try and catch problems so we can see what's going on Again, just trying to run these things on a regular basis. We recently started running the static analyzer might lay too So there should always be a latest version of the static analyzer output that's available Okay It's now on to the real interesting stuff, which is the actual status of compiling the kernel Okay, the first one I've already talked about a little bit and that is that LVM as of 3.3 Doesn't need any patches. Okay before that there were a few things we needed patched. Otherwise, it would it would crash with the kernel code or Spit out the wrong code for the kernel in certain situations the thing about the kernel code Is it pushes compilers further and harder? I think than any other code base All right, so I mean that this is the thing that was really interesting a bunch of the times when problems were found found with clang We would say well, it doesn't work with this code I know they're like well, we use it for all this other stuff. We've never seen this problem before The Linux kernel pushes compiler that hard Okay, so we found problems that nobody else has found even though it's used for other kernels All right so Upstream now works 3.3. There's still a couple things that we can't do we can't use the integrated assembler Which is a bit of a pain. We'll talk about that later. We can't compile boot code for x86 because we don't support 16-bit code unfortunately, I know We we've asked about that and unfortunately what it comes down to is the amount of work that will require to support code 16 is Sufficiently large that Nobody has the time to do it. It essentially there's only a very few narrow places where it makes sense. I agree with you Unfortunately It's not something it's likely to ever happen. So which really sucks. Oh, yeah And then of course as I said before we're getting a lot of support from the LLVM compiler developers whenever new problems come along They are highly motivated to fix issues that we are now finding in the Linux kernel That's right that we were finding when compiling the Linux kernel I should say and wherever possible. We're trying to get things Changed in LLVM as opposed to just changing the kernel, but there's I'm gonna show you some examples where we don't have that option Okay, so we're on to the channel challenges of actually building the kernel itself So the first thing is unfortunately different compilers have different standards that they that they support now GCC supports all the different standards to varying levels But defaults to GNU 90 now it used to actually default to GNU 89, which is a real standard well It's it's it's C 89 with a bunch of extra magic Okay, but I recently realized when I was preparing these slides that the standard recently changed and it's now called GNU 90 and GNU 90 is apparently GNU 89, but with a bunch of the C 99 stuff ported back again Okay, so it's this weird mishmash of several different standards that is becoming more standards compliant But it's still very GCC-ish Okay, but the problem we have unfortunately is that we now have clang We are now compiling the kernel with clang with the GNU 99 Because clang supports the 99 out of the box, which is essentially C 99 with a couple of guineasms Okay, versus getting 90, so we're having a few impedance mismatches between the code because GNU 90 sees the code in this fashion and GNU 99 sees it in this fashion Okay, now the good thing is is that GCC is slowly moving towards C 99 Standards by default, which is awesome because that makes it even easier for clang to work right to interoperate And in fact, I know of a couple of people that are doing this already because as we move to the next version of GCC It's becoming even more C 99 compliant And so they want to make sure that the Linux kernel code again follows C 99 standard as much as they possibly can That can only help both compilers The kernel also expects a certain functionality of this that is that is added specifically to GCC for the kernel and wasn't necessarily written down So there's a few things there that don't necessarily track and to a large degree the GCC isms that make sense The LLVM guys have added Where it doesn't make sense in other words, it doesn't fit into their architecture They've pushed back Okay, but the problem is if it's not written down anywhere There's no doc. There's no specification to write to and so they haven't been able to add those kinds of Functionalities they basically say if there's stuff that's not in the documentation. We won't support it. We only support document documented features it's also a number of GCC extensions and flags aren't supported and One of our frustrations also we found is that a number of the built-in functions are different there Ever so slightly different and sometimes hard to find What the differences are? But that's to be expected because built-ins of course are not portable So the first problem we had when we start looking at this is that k-build is very GCC centric Okay, so it makes the assumption that you're using a GCC or GCC like Rather using a GCC like compiler Amongst other things GCC will complain if you give it a flag that it doesn't understand it will say nope I don't understand this and fail Okay Clang on the other hand will say I don't support this but I'm going to continue Okay, so the problem is is you actually get to a point where CC option which is a macro that tries to figure out whether an option is Supported by a kernel by a compiler I should say Essentially breaks because as far as claims is concerned based on the macro it supports everything Okay, so unfortunately we had to put an if def into the make file that essentially looked at the warning output with grep to figure out whether or not That something was supported or not. I know it doesn't make sense They were trying to be permissive apparently So they're trying not to break a compiler the point is is that flags generally are a good idea not necessarily required There's certain situations, but that's not true but I Would agree with you on that So there are a number of people in the clang community that agree with you and me on that there have been a number of efforts to change that and unfortunately they've broken a number of Large systems outside of this and so a lot of people actually already depend on this Functionality and so the change hasn't been able to be made the default yet So people are still working on it and I hope it to be fixed one day, but right now we don't have a choice So unfortunately have to do this But it's only three macros we have to rewrite and essentially it's a pipe with a grep and that's all it is It's nothing too horrible It's hidden Yes, but They are run way too many times. Yeah. Yeah, you'll find that the the clang build of the kernel is still faster than GCC despite the extra forks Right, so we had to add patches specifically for K build The next one is name registers and this has been a bit of a problem GCC has this Quite nice extension that allows you to name a register as a C variable And so as a result you can use it directly use the stock pointer directly for instance, which is where we use it the most in the kernel So for instance next 86 we The the the code actually assigns ESP to current stack pointer in arm. It's sp. That's a sign to the same variable Unfortunately named variables are not something that clang can support and the reason is is because The code is up here The machine code generators down here and the AST is in the middle and that information isn't available above the AST Okay, this is something I've asked for several times and talk to talk to the various guys on the the list Unfortunately, this is not something they can support even for the stack pointer No, I'm still having this argument, but essentially what this comes down to is we have to Compile with clang unfortunately have to drop back to inline assembly, which really sucks Those of you who saw my patches this week We sent in a bunch of patches that that tried to get rid of this on the arm side Unfortunately, it incurs an extra assignment, which means that the code is very very similar except Yes, there's a push at the beginning and they pop at the end kind of sucks and an assignment So we're not entirely there yet. We're trying to figure out a way around that because obviously This potentially can happen a lot and all those extra instructions kind of sucks So but again, we've got a couple of ideas of how to get around it We might have to resort to an if def for now until it gets maybe we can figure something else out But we are making noise about this and trying to figure out if we can get this fixed But not not in 3.3. It's already it's already done The next one is something we refer to as place or variable length arrays in structs now VLAs or Variable length arrays is something that's supported by the by the standard Okay, a variable length array is an array that you don't know the length of it until runtime Okay, which is actually pretty cool You can do some neat things with that not everybody in the kernel community likes VLAs, but they do make certain things easier Okay, the problem is when you put them into structs So you'll see right here what we're actually doing is We've got a variable length array in a struct with something after it now. You can actually put a undefined Array at the end of a structure Okay, that that is allowed by the by the standard. However c99 and c11 Explicitly forbid you to put To do this so essentially have a variable length array in a struct that's not at the very end So the problem is we actually do this in a number of places It's actually used in the net filter code It's used pervasively in the crypto code and it's used in a couple other places like the gadget driver for USB Now in the USB case that's just used for convenience and then they're okay with a patch there The problem is is in the other areas. It's very pervasive. It it breaks a lot of stuff Unfortunately, we have to patch that out and that's something that that we're having some some pushback on unfortunately This is not something that can be added to clang Again, we've asked the problem is it would take major structural changes Inside clang that are detrimental to a number of other things that clang can do so it's not it's not a feature That can be supported and since it's expressively forbidden by the standards. It's not something that people want to touch Next one is nested functions Okay, there's only used in it in one place that I'm aware of but apparently it's people have found it in other places This one's actually in the think pad ACPI driver. I've posted a patch a couple of times I've never heard back from the the maintainer unfortunately It doesn't really do anything useful in that code. It's a fairly simple patch. It just shovels the code around a little bit Unfortunately, again not supported by the standard. It's not standard see clang doesn't support it Fortunately, it only affects think pad users. So it's That that use this this older driver. So Not a lot of people are too worried about it There are some other incompatibilities that go a little bit further a little bit harder though One of them is that attributes which again is a an extension a lot of the attributes are the same between the two Compilers, however, there's some slight differences that can cause problems When it comes to modules for instance, we alias the init and exit code All right, that's how it works when we're doing out of tree modules The problem unfortunately is that other attributes that are associated with those symbols are not aliased over at the same time So things like the link section Okay, so unfortunately in this case what we had to do to get it to work with clang as we had to Reapply in it and exit sections linker sections to those those symbols Okay, not a huge deal, but again, it's something that GCC is doing quite handily clang can't And so it's a matter of reapplying those those linker sections now. It actually still compiles It's just a matter of and you can actually load a module. You just can't unload a module and that's kind of It's not something that we It's not something we realized until recently so we haven't actually looked into fixing it yet I would hope so but right now it's such a small patch and it doesn't hurt anything that that we would like to push it up Stream and then un-patch it when we don't need it anymore if if that's acceptable We haven't we actually haven't had that conversation yet. So I mean that's that's part of it We've had a To be to be really fair. They've been very permissive. They've taken a lot of patches. They've made a lot of changes We aren't having as much luck on the kernel side so quite honestly we're getting way more pushback from kernel developers than we do from LLVM and What it comes down to what it comes down to is on the kernel side. We're getting pushed back That is reasonable. I'm not trying to complain here, but we're getting pushed back on Small things on LLVM. We're only getting pushed back on really major structural things on the small stuff They're saying no problems will help you So I'm not I'm not trying to point any fingers here I'm purely saying that There's there is a perception that that you just brought up the perception is I Don't think is necessarily fair for the for the small things for the the reasonable things. They're they are bending over backwards Okay, I'm sorry. I apologize It's evolving for the moment. Yes. Yes, of course, of course agreed agreed if you actually look okay, so so just to To repeat that so everybody can hear the the question is essentially that the kernel is written in a a Dialect of C that is essentially C with a few GCC isms for historical reasons The concern is that adding clang specific stuff can potentially Make maintenance more difficult. Is that is that fair? Okay? I agree if you actually if you actually look at the patches that we're proposing Most of them are very small innocuous and don't have major I At least feel that they don't have a major impact on the overall maintenance. They're very very small now in certain situations Like names registers. You're right. There are extra instructions involved and I would I would argue that that's not necessarily a good thing But we are trying to address those those issues we're not trying to make the kernel code worse We're trying to make it work in both cases as as well as we possibly can So we're certainly gonna work as well as we can to address all these concerns, but like I said as much if not more Help and and bending over back. We're just being done on the LLVM side So please don't feel that they're they're saying no they really want to get this to work No, no, I just I'm just making making sure we're all clear on that. So that's all Is that cool? Oh, they suck. Oh, no, no, we don't want to use depth. So we've barely used them at all We're trying very hard not to use them. I we completely agree Completely agree on that if depth they're evil the next one is actually to do again with the gunner 89 gunner 99 changes and they have to do with Essentially, the gunner 99 and 99 actually disagree on how how inline functions are done Essentially, you know 89 allows you to have two definitions one that's in line one That's not okay, but it doesn't the in line actually doesn't emit a function a an externable function, okay in C99 in fact, there is an externable function that's that's Defined at that point. So you actually have an inline version and a non inline version with the same statement the Appropriate answer here is to essentially change it into using static in line and whenever we've had this conversation from people ultimately The maintainers involved said, yeah, absolutely make it a static in line So we've got a few places that we've done that already in the x86 side actually Greg Greg Crow Hartman helped us with those We have a number of more that have you done on the arm side that we still need to make changes for it is the right thing Yes, yes right This is essentially doing the same thing Sorry in these particular examples that we're talking about static in line does the right thing in both cases for both compilers I mean, perhaps we can talk a little bit more about that afterwards Okay, the next thing is there's a couple of places where we have Very complex statements that ultimately a clang chokes on so in this case we can see there is an alignment statement there That's relatively complicated. We merely broken into two statements It actually I think it actually makes it a little bit easier to read essentially does exactly the same thing both see Both GCC and clang actually end up doing the same thing with that particular code Okay, the big thing is of course, unfortunately that the integrated assembler actually can't be used and again that has to do with code 16 and in the case of arm the problem is the inline assembly isn't actually in unified format It's in the pre unified format using a lot of new extensions unfortunately the the the IA guys have tried to add the extensions unfortunately There's agreement between both the GCC and the the clang camps that it's a bad idea to introduce those GCC and is in there the pervading Thought is that we should really bring all of the assembly code up to using unified assembly format language Okay, which is the same language is a same format rather. That's used across tools Unfortunately, this isn't something it's easily fixed again Unfortunately, it means making changes to all the inline assembly in the kernel for arm To bring it up to a standard so that IA can be used For the moment we just turn IA off and we just revert to using the GNU assembler But it means we slow down there's more forks during the build and it means there isn't as much error Fix it hints that you can be done added Okay, right now. We're down to this number of patches We have patches to the general kernel mostly k builds we have 18 of them and again They're just you know tiny little patches. We have our 11 that are specific to arm six that are currently specific to arm 64 And we're down to eight for x86 64 Okay, so what's left to do? We're mostly upstreaming patches right now There's still a bunch of stuff that needs to be tested and fixed We don't claim that everything is working although we've got a list of kernel options that are currently broken on our website We still have a number of section mismatches and merge global issues that are being reported We haven't figured out whether they're necessarily causing problems yet. We have only in the case of the init and exit code Okay, but ultimately We need to track those down a little bit more Okay, and finally of course enabling IA would be really awesome. It would give considerably better Code coverage and allow client to work that much better How can you help? Well, I've already said a lot of those kinds of things come join the project report bugs to our patches Help us get our patches upstream Submit new patches all that good stuff and I know I rushed the last couple of slides But we're essentially out of time any questions that you guys want to have I can stick around afterwards to talk Because we're into lunch now, but otherwise Let's let's do some quick questions here. Yes Run time performance. Okay, people always want to know is it faster and in certain situations the pharaonics tests have shown that general code is Similarly fast in some cases faster in some cases lower We haven't done the same sort of thing on the Linux kernel because profiling the kernel in general is difficult I'm not saying that can't be done. It's just that we haven't spent the time on it I imagine that it is similar I would probably err on the side of being a little bit slower because there's certain optimizations. We can't do yet Okay, but it's it's close I wouldn't I wouldn't pretend to say we're outperforming GCC yet Yeah, Mark as well So so being able to compile the code base and analyze the code the code base may prove valuable in and of itself independent of running the culture we actually have several people on the mailing list that are actually doing exactly that they are taking the Linux kernel code they're Compiling it to LL IR or to the AST Pastry and then and then actually evaluating how efficient it is It seems like that's a very popular Academic thing that people are doing right now, which is which is kind of cool. Yeah, if you are interested in You know making the performance argument. Yes Something we keep asking the GCC folks for and never actually get okay is we the Linux the the kernel is by all More or less more or less a one large event handler It is absolutely it runs a lot of it runs cash cold much more often than you would see user space code true and For that reason code size matters luck more in the kernel than it does the rest The problem is if you compile minus OS right GCC assumes that you are basically trying to squeeze into a ROM and That you're trying desperately to do anything you possibly can to squeeze that code into that ROM because you can ship your product otherwise and so we end up produce it produces code that is Faster 80 percent of the time right, but the remaining 20 percent more than make up for it, right? so we would really like to see something that is Is have an optimization mode that is heavily biased for size But isn't completely stupid about it, right? Well, I can't talk directly to that. However, I've played a little bit with OS and the various O levels When comparing actually the place code both in GCC and in clang and one of the things I found interesting is that o3 and o s actually are very similar in clang and they both tend to be smaller Code size-wise than the GCC stuff so very narrow example But I found it interesting that OS and o3 seem to be the same in a lot of situations. So Maybe Maybe that that might be the case. I don't know I can't say one or the other But I found that kind of interesting they were the same size Any other questions? Yeah T-shirt working at a t-shirt I've got stickers with me t-shirts this this I mean certainly if you want one they I got this printed at One of those online the things I can certainly put up a store if any people want one. Yeah, this is our logo So it's the obviously tux with the the Clang compiler wings on it. So Thank you. Anybody else any other questions? Okay, anybody wants a client sticker. I've got some up here