 And basically, I'm now the project lead of the project, bringing together a bunch of different people to compile the Linux kernel with Clang. And the first thing that I tend to get asked a lot about is, you know, what is Clang LLVM? People get confused as to what the tool is and everything else. And really what it comes down to is LLVM is essentially a tool chain toolkit. It's a way of building other things. You can build lots of different things like compilers, like Clang, you know, linkers, just-of-time compilers, but there's also such things as source code analysis tools, like static-camp analysis tools. There are tools for extracting other metadata from code, code refactoring, and some other things, sorts of things I'll talk about. You can also tightly integrate it with IDEs. It's also a tool chain suite. So, again, we're primarily going to be talking about Clang today, which is a CC++ Objective-C compiler. There's also, we've probably heard of LibC++. That's the LLVM project's re-implementation of the CC++ libraries. Compiler RT is sort of like LibGCC. It provides capabilities that may not be available on a particular processor. There's a static analyzer, debugger, a couple of different choices for linkers, and a bunch more stuff that's part of the main project. Now, the really big question that I get asked all the time, certainly, by a lot of people here at this conference was, you know, why would you want to use Clang? Why not something else? You know, why not the more traditional choice? And so we're just going to talk about that first. Why would I want to use Clang? The first thing is that it's a fast-moving project. You know, tool chains tend to be slow-moving projects. They tend to be something that you need to make changes over a lot of different areas and everything else. The reason that a lot of people like using Clang is that if there's changes that they need to make, it's something that they can get in relatively quickly. You know, the source code is easy to follow. Test cases are abound. Basically, you can't add a feature without adding a test case. The community is really helpful. It's across a lot of different areas. If you jump on the RC channel, you're going to get help with whatever that happens to be. And, you know, it's the kind of thing where, you know, you're getting a similar size and speed of resulting binaries to something such as GCC. When I say similar, you know, sometimes it's bigger, sometimes it's smaller, sometimes it's faster, sometimes it's slower. Statistically, they're really, really, really similar. Fast compiles. This is one that's actually really cool. We're all living a compile test, you know, debug fix loop. And the faster your compiles go, the better. I spend a lot of time optimizing my compiles with things like Ccash and whatever else. The nice thing about Clang is generally it can compile your code faster and use less RAM in order to do so. And that's pretty exciting. The other thing is that some companies worry about having more than one tool chain across different kinds of domains. These days there are companies that deal with code on DSPs, GPUs, CPUs, different kinds of virtualization technologies and so on and so forth. You know, they've got things like DSPs driving their cameras, their audio processors, video processors. You know, maybe they've got stuff running on the GPU with CUDA. Pure on Android, of course, there's things like render script and so on, kernel, user space, applications, documentation generators. These are all areas where you often have different tool chains, certainly in the past. The nice thing about LLVM and Clang is it's the same parser, the same backend libraries that basically runs across this entire system. As a result, if you want to write an extension to the tool chain, all of a sudden you get it across all those different areas as opposed to if you're using disparate, even open compilers, all of a sudden you're either having to port or re-implement that same extension in a lot of different places. And then there's, of course, all the qualification, maintenance, you know, and so on that comes along with that. So having a single tool chain across everything makes an awful lot of sense. I hate to say it, but especially when it comes to embedded license is one of those things that people worry about. And the thing about LLVM is it started at the University of Illinois and it has a BSD-style license that everybody follows. And one of the nice things about that is because it's a library of functionality, it's embeddable into other things. And so you can basically take the parser, any of the code generation bits or whatever else, and you can embed it into whatever your project happens to be, whether it's open or proprietary, it doesn't matter. We all want open extensions, but if for whatever reason what you're working on is something that needs to be closed, again, it's compatible with this. The other thing that's really good is that quite frankly, as big as the open development community is, the closed community is also there, and as a result, you get a very, very wide set of people using the same sort of thing, and you also get a lot of different full-time people that are actually making it better at the same time. In fact, as many of you will probably know, Clang is primarily supported right now by Apple. And guess what? They're paying people to make this compiler better, and as a result, we gain the benefit from that in the open community. Questions over here? To be quite honest, most people have been actually happy about that as being a choice of license. The reality is that anyone who's used GCC in the past, very few of us are actually using mainline GCC. It doesn't matter what platform you're on, you're usually using some kind of leaf or some kind of branch, and sometimes a really disconnected branch of GCC. It still looks like GCC, but it's something that was GCC at one point, and it's now GCC with a bunch of stuff in it. Whether it's really kick-ass and awesome like a code sorcery, for instance, or whether it is, it's a really, really good branch of GCC, but it's not mainline GCC, right? And that's the reality is that in most cases, you get a compiler with patches that are never sent back upstream. The interesting thing about this particular compiler is there are probably similar kinds of things. The reality is, though, is that a lot of stuff makes it back upstream. It doesn't matter whether it's embedded, HPC. This is being used on some really, really big clusters here in the United States that I know of because one of our team members has access to that kind of technology. You have people working in embedded, mobile, all sorts of different areas, and in most cases, like on the Linux kernel, it's much easier to get stuff pushed back upstream so you don't have to deal with stuff than others. I haven't seen it. And the fact that the code base is evolving so fast, stuff is making it upstream really fast. And that's a good thing, so that helps everybody. Sorry. Another question. Right. No, very true. That's a good way of thinking about it. Yeah, and of course, even in the case of BSD, in many cases, we talk about BSD-style license because very few projects only use the BSD now. In this case, it's another BSD-style license. But, yeah, it's a blessing and a curse, and I've heard arguments either way. All I know is that in practice, this seems to work remarkably well. And one of the really interesting things is that you've got embedded companies that are traditionally more concerned around licensing issues as soon as you're pushing code into a BSD-licensed project, it seems like people are less worried about that sort of thing, which we all realize is not necessarily a real concern, but it's a perception thing and it helps. So the other thing is, and we used to have a lot more slide like this when we presented this kind of a talk at other conferences like at LPC last year, we used to show why Clang has lots of better things than some of the alternatives. And one of them was his fix-it-hints, and one of the things that if you've been using things like GCC for many years, we all know that a lot of the error messages that came out of GCC were a little hard to follow. You'd have to know what they meant, right? Sometimes you'd have an error message and it meant that there was a semi-colon somewhere up higher, but it wouldn't tell you necessarily where. Clang has always done this thing where it's basically shown you what the problem is, what line, what character, and how to fix it in most cases. So that's what these fix-it-hints are. Now, while preparing for this conference, I went through and did a lot more research again because one of the things that we were, the complaints that was made last time is the comparison between Clang and GCC that is done on the LVM website is actually between GCC 4.2 and the latest version of Clang, which isn't fair. So the interesting thing is that the GCC guys have actually gone through and redone the tests. One of the things that's quite fascinating is that GCC 4.8 actually does a lot of similar trends of things. It does macro expansion properly. It does the fix-it-hints to a large degree too. It doesn't do color syntax highlighting. Okay, fine. But from a large perspective, this is one of the things that Clang did much better than other things. GCC has caught up. So this is one of those examples where GCC has actually been pushed by the competition of another tool set to actually get better, which I'm really excited about. I always thought that was really great. The other thing that Clang and LVM bring to the table is static analyzer. Now, there are lots of static analyzers out there. First thing everyone says, yeah, but there's sparse and there's this and there's that and everything else, right? Yeah, that's fine. All of those things use their own parsers. They have to reimplement a C or C++ parser, right? They're not quite the same way. They're not actually compilers necessarily. In many cases, they're pattern matching mechanisms. In this particular case, it's actually using the same front-end parser as the compiler to go to a parse tree to do the appropriate work and then to figure out what's going on. In this case, you can actually see it spits out HTML. So one of the nice things about running Clang or the static analyzer checker on your code is you get a whole bunch of HTML files that then show you different kinds of things. In this case, you can see that if OPTARG is null, for instance, you can tend to go through to a point where you're calling a string to UL with OPTARG as null just because of the test at the top, right? Where you say if OPTARG and string case camp, well, if OPTARG is zero, you're still going to fall through and do the wrong thing, right? That may not be entirely obvious. Checker can find all these things. It'll walk all the different paths and figure out what's going on. There's a lot more examples. That's just merely one of them. There's also a bunch of other things that are going on. If you actually watch a lot of the videos that are coming out of the LLVM development projects, there's some really neat things like Google, for instance, is doing this thing with their compiler team where they actually have a huge compilation farm that recompiles all of their code every time they make a change. They don't do incremental builds. From scratch builds every single time. The guys that build this stuff and maintain this stuff thought it would be really neat to use LLVM alongside GCC in order to get the better error checking, and that's what they do. They actually use GCC and LLVM at the same time. But they thought, well, wouldn't it be neat to go a step further? We have this parse tree that comes along. Why don't we use that to actually look for problems in this sort of thing? There's a great talk, and I encourage you guys to go out and look at it because it's impressive in its own right. But they actually have a mechanism where they actually look for problems. And then through the same kinds of systems that are used for fix-hits, they actually spit out new code. So C++ goes in, new C++ comes out that has the bugs fixed in it that they know of it. Every time they find a new class of bug, they write a new filter that basically fixes this stuff. Email goes back to the developer. Developer looks at the code, submits it, problem solved. When APIs change, when common problems come along, and we've all seen this in the kernel, right? You go through, you fix a couple of things, right? But it's still broken into a bunch of other different places, right? The whole point is that in this particular group, what they do is they write an extension to their build system, and then it starts getting fixed everywhere else throughout their vast number of lines of code. And I would love to have that kind of mechanism for the kernel where there are kinds of large code bases like that. The other thing that's really nice is things like the Static Analyzer. It has plugins for various kinds of things, like malloc free-checking, right, to make sure that memory's always free. You don't actually start leaking memory. Static Analyzer similarly can be extended. And in fact, people have written things like K malloc and K free extensions, for instance, and so on. But there's no reason you couldn't do the same thing for locking kinds of issues or what have you, you know, following different paths, trying to find kinds of issues that are there. So there's actually a number of different researchers that would very much like to be able to use the Linux kernel with Checker to be able to do research around looking for these kinds of common problems using Static Analyzer analysis tools. So an awful lot of people are already using LVM and Clang technologies. I just chose a few here. There's a whole bunch more of them. But certainly on the open side of things, most of you probably have an Android device in your pocket. Every single one of them has LVM on it already. That's in the form of render script. You know those dancing backgrounds, those live backgrounds? That's actually running on your GPU. That's actually code that has been compiled to LLIR that is then taken through a JIT that is in fact LLVM to take it to GPU code at the last minute. So your CPU can sleep while your GPU is still updating the screen. Okay. So that's actually something that's already in Android. It already works on ARM MIPS and X86. You then have the Android NDK, the one that was just released in the fall. It actually ships Clang as one of the supported NDK compilers, native compilers. It's also something that's used very in a very central role in, for instance, the Gallium 3D project. There's LVM Prite Driver. There's Clover, which is part of OpenCL. It's also used in a lot of things like shader optimizers and so on and so forth. Very, very heavily used when it comes to doing graphical stuff on GPUs and so on. And then the quintessential place I go to see just how far it's come actually is is Sylvester LaDrew. He's part of the Debian project. And he has an ongoing thing where he tries to recompile everything in Debian using Clang instead of GCC. And every year he seems to get a little bit further along. The interesting thing is that at Fozdom, he just gave his latest talk. In fact, this is one of the first times that fewer packages actually compiled. And the interesting thing is this. When you look at the numbers, in fact, more packages were fixed, more code was fixed upstream so that it didn't break, but more checks, more warnings were added to Clang and, as a result, packages that used to work actually started to break. So the interesting thing is that a number of upstream code maintainers are actually starting to look at Clang and actually starting to look at the kinds of things that are coming out of that and make better changes. It's just that it's a bit of a moving target sometimes. But we're still on the order of 10% to 12% of Debian packages that don't build. And when I say 10% to 12% of the packages, I mean with no extra help. Basically just passing and use this compiler instead of that compiler. So currently Clang is actually being commercially deployed in a couple of major places, obviously in Android, as I just said. The other one, of course, is as part of Xcode. So pretty much every iOS and OS X system right now, all their code is universally built using Clang now. Just as a bit of a short list, it seems to me that one of the nice things, again, because of this competition side of things, is we actually have better macro expansion, better reporting, better fix it hints, and the address sanitiser that was added to GCC. All those things have very likely been pushed forward by the competition from the LVM and Clang side. So there's a net benefit to both communities from the inclusion of these kinds of technologies. So again, I'm the project lead for the LVM Linux project. Just want to tell you a little bit about that and what it is. We're actually a meta project. So we're not part of the kernel community or the LVM community. We live between those two communities. We have people on both, on the project that are in both communities. We're just trying to get one to work with the other. That's it. We don't want to maintain our own tool sets. We don't want to maintain our own kernel, just patches. So it's a matter of building those patches, testing those patches, and then getting them upstream to their respective projects. And of course, the big thing for us is we're trying to bring together like-minded developers because there were a lot of different people working on this in the past. We're all under the same kind of umbrella. The work in the past was done by people like Bryce Lelbach, who originally announced this kind of work at Collab Summit a couple of years ago. Edwin Torek, Sidatlit Dilek, and PAX team, where some of the original people did a lot of the work, along with Mark Charlewa, who did work on the side as well. And they've all been joined by me as well as by people like Ben Simone Mahler and the sinious Tinti, who is a fellow from Brazil, and I'm just singling him out for a second because, in fact, he helped me tremendously preparing for both this talk and Android Builders. He basically spent an awful lot of time helping me with the port of some of this stuff to a number of different Android targets so that we actually had kernels that work on real platforms. His was the work that actually got our patches working on the Nexus 7 tablet, which was really interesting. We've got a wiki page, lvn.linuxfoundation.org, that has things like bug trackers, roadmap, different how-tos, and so on and so forth, sort of the quintessential project wiki. Our system is basically an automated build framework because a git tree there you can get, which you basically go into, and it'll just build the world. It'll download all the source code, all the patches that are necessary, tool chains that we need beyond that, linkers and such, the kernel code, QMU and LTP and that sort of thing. We actually maintain patches for all those different systems such that we download the code, patch it, build it, run tests. It's actually very similar to what I just learned, the way BitBake works, but we were doing stuff before we understood that particular tool. We actually currently do our patch management using Quilt on top of either Git or Subversion, depending on the project, and for various reasons, not only to make sure that our patches still work under GCC, but also because, in fact, we still have to use the GNU assembler and linker for this particular work. We actually still need to have a GNU cross-toolchain at this moment behind the scenes, and in fact we use the code sorcery compiler by default, but you can actually choose between that, Linaro and Android in order to provide those pieces. I'll explain in a second why that is. Actually, I'll explain now. In the case of the assembler, LLVM actually has a built-in assembler called IA, or the integrated assembler, which is very fast. It does have some issues around the inline assembly syntax, which is different from what's used in the Linux kernel. As a result, at this moment, we've actually turned it off because it's a lot easier to get what we need to have done with the GNU one. In the case of the linker, linkers are hard to write. Quite frankly, the two linkers that are part of the LLVM project right now, they're really young and they're not ready for primetime, is what it comes down to. The fact is that the default linker in GCC and, of course, gold, those are the two linkers we tend to use a lot. They work really well. Inclined or compatible, it's no big deal. Incidentally, you can choose which toolchain you want just by passing it as a make variable. I honestly don't know. I don't do work there, so I honestly don't know. The fact is there's two linkers right now. LLD is the one that was started about a year ago and it's meant to replace the one that's on the beginning side. The other one is called MC linker and it's actually designed to do linking on target. It's a very, very small, fast, though not as capable linker. I have a feeling I know why it's being added probably for just-in-time native linking, but it's one of those things where everybody's doing a lot of different work for their own reasons. As far as our automated build system is concerned, it's all very self-explanatory. We've got our own help system and such, but basically you just choose the target you want to build for CD in there and you have all the different make targets you need to make the appropriate things happen. All the build dependencies are there. If you just type make by itself, it'll download the appropriate things, build the compiler, build your patch kernel, and then if it's supported run it under QMU to see whether it actually works. Our current supported target lists and a couple that are upcoming is there. Most of my work at least is done on Versatile Express inside QMU, but we also, because of one of our team members working for Qualcomm, we also have support for Qualcomm MSM code on top of 3.4. Our x86, 64, and i586 code is all mainline now, which is something that only happens since our last talk. The Raspberry Pi is fully supported on 3.2 other than we're having troubles with the USB driver, so we're having some issues there. We still haven't been able to figure out as a result ethernet doesn't work, so bit of work in progress, and again we're just about to, our guys just about to rev that up to 3.6. Raspberry Pi of course is not mainline, so it's the best we can do. On the Nexus 7 and on the Galaxy S3 we're actually using the kernels from Sanogen Mod, so we're stuck at their particular versions of the kernel. And the Beaglebone, I was in a talk from Matt Porter at ELCE and started hacking the Beaglebone code. I'm sort of most of the way there, but it needs a bit more love before it's going to work. But ultimately those are the systems that we're currently using it on, and the top ones all work. We also have the BuildBot that's continuous integration for all code bases that we deal with. Every time Linus does a check-in into the Linux kernel code, we build and retest. Every time the guys that are working on Clang and LLVM do a check-in code, same deal. So we know as soon as possible whether this stuff breaks. We live on tip on all of our, all those source code trees, so when stuff breaks we have to fix it then and there. Yes? Yep. Primarily ARM in X86, we do have ArchDirectories for MIPS, but quite frankly the people that are currently working on a project have their primary focuses on ARM in X86. So it's not that it can't work on something like a MIPS for instance, it's just a matter of nobody's stepped up and done the work yet. We would welcome people joining to do that, to help us on that work. There's always more work to do. Like I said we've got a BuildBot that does all the stuff for us, and basically for instance our LTP tests that run at night, unfortunately if you go to our web page right now and look at our BuildBot you'll find that our LTP tests are actually failing right now and that's because the SD driver for the kernel under QMU broke about a week before I got here and I haven't had a chance to fix it yet. So this is the kind of things that we know on a very regular basis what is stuff working right now. So we have a mailing list and an IRC channel that we hang out on to coordinate all this work. We're on three continents right now, so many, many different time zones. So our scheduling meetings is kind of an interesting challenge. As far as actually building the kernel's concern, there's a number of different challenges that we came across. The first one is that Clang is a unified cross-compiler. Instead of having a bunch of different versions of CCC depending on what your architecture is and what your GLBC version you're using or whether it's bare or whatever, Clang is a single tool chain. So you just pass on the target you want to compile for. So the first thing is finding the right triplet for what you need to do. That was actually one of our first problems that we had. The second one again is we can't use integrated assembler everywhere. We got it to work on x86 largely, but then we had to turn it off because we had problems with syntax. The other problem though is that IA is 32 bits only. So we can't do, for instance, a 16-bit start-up code on an I586. Again, we have the dependencies on the GNUAS and LD, for instance. The good thing, however, is in supporting all those different compilers, in fact, Clang has a dash GCC tool chain option that actually allows you to choose where it is that you want to get your compiler from when it comes to the back end to get those extra bits and pieces. But there's a question over here. A triplet is what you see at the beginning of your cross-tool chain. Things like ARM, Linux, GNU, ABI with dashes in between. In a lot of situations, it's actually four words. Strictly speaking, you'll often see ARM, non-Linux, whatever. The fact is that a lot of people only use three of them in the LLVM parlance that's called a triplet because there's three of them. That's all it is. Just a way of specifying your architecture, your operating system, and then your back end by ABI. We also have problems because the code base itself has a number of GCC dependencies. Linux kernel was developed with GCC by people often sitting at desks next to GCC developers. There are a number of GCCisms. On top of that, the kernel has always gone beyond what the C standard has allowed at that particular point in time. It's a pretty complex code base as a result. That makes sense. One of the things that we see is that GCC actually defaults to the GNU89 standard. By standard, I mean what evolved into what is the C standard that GCC uses. Clang does support GNU89, but it also supports the newer standards, and actually it defaults to GNU99 by default. Right then and there, it's a little bit more stringent than GNU by the GNU tool set by default. It's not that GCC can't do the same thing. It's just that it doesn't by default. And so therefore, we don't use it that way. The other thing is that the Linux kernel actually also expects some undocumented GCC behavior. Some stuff was added to GCC and was never actually written down in the manual. It just works magically. That's been a problem chasing some of that stuff down. The last thing is that GCC has a number of built-in functions, and there's a couple that we rely on in the Linux kernel code that have no equivalence in Clang. Yes. I'm going to get that in a couple slides, if you don't mind. Okay. The other thing is GCC is GCC specific. For instance, when it comes to C option, right, whether or not an option is actually supported in GCC or not, GCC does the, I think, sane situation of if you pass in an option, it doesn't make sense. It fails. It returns an error. On the other hand, for various reasons, decides that it's merely a warning. So it will merely say, I didn't use this option. But the problem is that C option requires, it actually relies on the return code in order to figure out whether that option is valid or not. So unfortunately, we need a Clang specific version of C option. And there's a couple of different K builds, macros that are like that, that we had to actually patch. Yes. This is an autoconf. No, this is in the kernel. This is part of K build. Similar kind of idea. Similar kind of idea. There's also a whole bunch of different options that go to GCC that currently aren't supported in Clang. And some of them probably aren't. Now, if you want to know exactly what they all do, we actually talked about them in the talk that Mark Shalabua and I gave at plumbers last year. The slides are online. The link is in the slides here. Yeah, you can look at more information there. So one of the undocumented features of GCC to answer questions here is something that we refer to as place or variable length arrays and structs. Now, VLAs or variable length arrays are something that are used very heavily in the Linux kernel. And there's no problem with that. They're completely supported in the latest C standards. So unspecified arrays, no big deal. As soon as you put them into a struct, there's a problem. And the problem is that it may not be the last thing in the struct and you may have more than one. In fact, we have code in the Linux kernel that actually uses structs with more than one unspecified or runtime-specified, I should say, array in a structure. Now, it's done for reasons to make the code easier to read and so on and so forth. The reality is, though, is that this is an ADEF functionality. This is not something that actually the C standard allows. In fact, if you look at the C standard, it explicitly says you can't do that. This is actually a feature that's not documented that's explicitly forbidden, and yet it's something that we rely on in a number of situations in a number of subsystems. And quite frankly, it's not necessary. And it precludes the use of other compilers. It's actually used in a couple of places. IP tables, it's used in all the HMAC functions in the crypto subsystem and the gadget driver. Now, we actually submitted some patches upstream. We've had some pushback. The nice thing is, at least on the USB gadget driver subsystem side, they've more or less said, well, we don't need to use place. We're going to rewrite it to use something else. They didn't like our patches. That's fine. They're fixing it. So we're pretty happy about that. The other one is nested functions that she didn't know there was nested functions in the last kernel. There is. It's in the ThinkPad ACPI driver. I just actually patched this out and sent a patch upstream, although I haven't heard anything about it since. But this is one of the places, for instance, that we can't actually get this to work. Now, this is documented. GCC explicitly supports this in its documentation. But again, this is an ADA feature. This is not something that the C standard actually allows for. And unfortunately, it's something also that certain architectures need to go through a number of different hoops in order to get it to work appropriately. So it's something that a lot of kernel developers actually have forbidden when patches have come in. They've asked people to rewrite stuff. Let's go over here first. I've sent a patch to the Linux kernel. Nested functions is not something that the LLVM projects, along with variable length arrays and structures, neither of those are things that they will support. So both nested functions and variable length arrays in structs are, in fact, things that made it into the GCC tool chain through the ADA port, the ADA front-end. So it's leaked functionality. Now, I can't verify that for sure, but from what I've seen and what I understand talking to various people, it's functionality that made it into the C code unintentionally that... Yes, it is, as we understand it. What else here? Oh, yes. The other thing is is that there's functionality like explicit register variables. Unfortunately, those are also not supported under Clang, at least at this point in time. So the ability to name a register by name, for instance. So there's an x86 example, there's an ARM example. In both cases, we've had to change that into inline assembly calls instead. And then there's other things like, for instance, if you look at the bottom line on the slide there, that is code that GCC is more than happy to compile. Clang is not. And what we found is that we had to split it into two lines. I think it's a bit more readable, both GCC and Clang as a result to exactly the right thing with that particular line. Unfortunately, then we get some incompatibilities. So these are actually issues that LLVM has, not doing the right thing. In this particular case, we get a lot of things called segmentation reference problems. What we actually found out, again, just actually before I wrote these slides before I came to the conference, we were thinking that LLVM was in fact putting sections into the wrong place. It turns out we just found out that in fact the CPP stage in Clang is actually dropping certain attributes. So in fact, they're going into the wrong sections because init and exit are being dropped. So this is actually something we've just discovered that we need to fix. So it works. This is one of those areas where GCC is doing the right thing. But it's also one of the areas where you can see that GCC was actually being pushed really hard by the Linux kernel code. And other compilers that haven't been through that same trial by fire haven't necessarily managed to catch yet. But this is going to solve what you're going to see if you try compiling our code right now. You get a lot of errors about merge global problems. Basically, once we fix this, all that is going to go away and work properly. The other thing of course is assembly, inline assemblies handling. Right now the standard that's used in the kernel code is GNU 89. Again, that's something that is not used by Clang. They're using the newer C99 standard. Again, as a result, we have to use the GNU assembly at this particular point in time. And then here's one of those situations where we're talking about there's a built-in constant P. That's actually a compiler built in. That's not something that Clang does for whatever reason. There's a piece of code where that is whether or not a piece of C code can be optimized away under the situation where a pointer is passed in that is a constant. It seems reasonable enough. Unfortunately, it's not something we can do in the same way using Clang. So currently this is an example of something where we actually don't have a solution yet to be able to push upstream. So the status of actually building a Linux kernel, basically how close are we to, how many of these things have we fixed, and how close are we to actually having full cable support? We actually now have full cable support. We have patches that actually allow you to use cable without wrapper scripts, which is what we've been doing fairly recently. We actually also have fixes for all the explicit assembly uses. Again, we've moved over to inline assembly calls. We do have patches that do the removal of variable length arrays and structures. We're just getting a bunch of pushback on that side, so we're still trying to do what we can in order to do things to get that upstream. Although I have to say props to the Yocto project, I was told that they will take these patches into the Yocto tree, so I'm pretty happy about that. There's also the use of extra inline in a few places still, but most of them were taken out by Greg Crow Hartman last year. In this case, there's still one that's in the ftrace code in the ARM arch. Early when I was talking to Steve about it, he pointed out that in fact it wasn't his code, it was something that made it into the ARM. That makes it my problem, so okay, fair enough. We do have a workaround for built-in constant p. It is not something we can upstream, unfortunately. Then, of course, again, we've basically redone that weird aligned attribute cast into something that's a bit more readable, but that compiles on more than one thing. In the past, we had to actually rather heavily patch LLVM. The good thing is that most of those patches have made it upstream now. In fact, the biggest one of those patches was actually 64-bit type handling for ARM. That was something that wasn't supported until just a week and a half ago. We actually had a lot of help there. The nice thing is that the Qualcomm Innovations Center, this is something that actually affected them very quite a lot rather. They've been working on that really hard, and I believe they took their solution and they had to break it up into six patches. The last one just made it upstream. It means that LLVM 3.3, when it comes out, is going to have full 64-bit type handling on 32-bit ARM, so that's actually pretty good for us. What it means is that, in fact, currently we have no extra patches necessarily on the ARM side. Right now, the tip, the SVN tip for LLVM and Clang fully support the Linux kernel in the situation. We have about four left on the X86 side. Most of them are cleanup. They're not strictly necessary, but they solve certain corner cases. We still have some other issues. Like I said, there's still the attribute stripping for init and exit. We have to track down. That's not there yet. I almost skipped IA. We still have to get the point where we start using IA properly, but again, that's a syntax issue. One of the nice things Clang 3.3 is probably going to be usable when it comes to building the Linux kernel without any modification. I'm really hoping that prediction comes true assuming that we get the one right before fixed and in before 3.3 is released. What's left to do beyond that? Well, like I said before, we've got to get IA working. A big piece of work is we've got to get the blaze patches upstream. There's the segmentation linkage fix. We've got to track down and fix. Oh, and huge type of there. Lack of built-in constant P. And the final one is actually getting checker to work with the kernel. Try as I might. I cannot get the static analyzer that we have in LLVM to work with the Linux kernel. And that really is frustrating right now, because that was one of the things that I was really hoping to have working for this conference. Works great in user space. Can't seem to get it working on the kernel code yet. No, right now, basically it's it works by basically working on top of make. It subverts make. And unfortunately right now it's not understanding what K-Build is doing properly. So I haven't entirely figured out why yet, but basically when we run it, it spits out nothing as if everything's great. And of course we all know that that's probably not true. No offense to anybody in the kernel. So how can you guys help? If any of you are interested, how can you help along with this particular thing? Well, there's unsupported features and bugs that's only fixing. We have a bi-weekly Google hangout that we actually discuss some of these things and what works coming up. You can review patches that we've got there already in the project. Help us get those upstream. We're always looking for new targets, new arch support. Again, we're wanting to have as broad functionality as possible, so the more targets we can get, more test targets, the better. You can download the code and give it a try. Although we haven't actually published the ROM for the Nexus 7, the kernel itself, if you then want to add it to your own Nexus 7 is already building inside our system here. It's all good. The other thing is it's helping us with test cases and reporting bugs. Basically, we still don't feel we've got the level of validation we need. We really want to get to the point where we can basically say these patches work, they don't break the existing GCC compile, and they generate perfectly good Linux code. It seems to work. It's been stable. Certainly on my Nexus, on the guys who are using it on their laptops already, on the Raspberry Pis that people are playing with, seems stable, but actually proving that, we would love more test cases in order to have a better idea of how that works. And just to leave you guys with something, basically, who wouldn't want to have a penguin with dragon wings. So, thank you very much.