 Next up, we have Carlos O'Donnell speaking to us about securing GCC and G-Lib C. Please take it away. I really like Joshua's talk. I mean, we could keep talking about that stuff. I'm always interested in Yachto. I've done embedded stuff previously. Hi, everybody. My name's Carlos O'Donnell. I work for Red Hat. I work in platform tools for Red Hat. Platform tools is the broader team that provides C, C++, Rust, go system tool chains for all the builds that we do for Red Hat Enterprise Linux for the layered products above those things. The team is quite big. The talk that I'm going to give, and I'm going to have to start sharing my screen here in just a second, is here we go. The talk that I'm going to give to you was it was a team effort. Lucy Kerner, who is with us for Security Global Strategy and the platform tools team, how we put this talk together. So just making sure that everybody can hear me and everyone can see my slides right now. Someone give me a yes, we can. Yes, we can. Thank you. Excellent. So we're going to talk about securing GCC and G-Lib C. We're going to start. We're going to look at the past. We're going to look at the present. We're going to look at the future. We're going to look at some of the foundational principles that come out of us looking at the tool chain, at the behaviors that developers have in the past. And what we can do to secure GCC and G-Lib C, improve the security of the applications that you're developing. But before I do that, I want to start with a special thanks to Sudesh Boyarekar, who's on the platform tools team for security strategy. I also want to thank David Malcolm and Martin Seabor, who are part of the GCC team in the help of some of these slides together as we as we look over past present and future. So let's start by looking back at GCC and G-Lib C security. And I want to look a bit about the past practices in developing core runtimes and developing the compiler. And we'll see how they influence the kind of choices that we want to make, the kind of foundational principles that I think the new tool chain moves towards. And you'll see why those are important and why they're relevant to our future projects and why we would pick those. So just looking at G-Lib C, as you look back, and I've covered here a couple of points here, from G-Lib C 218 to 226, over the course of the years, we've had a number of these CVEs. And the thing that we see in these CVEs, the thing that's really relevant about these CVEs that happen in G-Lib C is that they are common defects. They're common defects that we see industry-wide. And that means that from a core runtimes perspective, we began seeing that we needed to adopt a more interactive model with the compiler, with the runtimes, working together in order to shift towards prevention. Because as I look back at these CVEs between 218 and 226, we see things like in the Get Outer Info CVE, we see integer overflows, we see stack and heap buffer overflows. In the Get Outer Info one, what's interesting is it's a pointer manipulation problem there. And in many of these cases, GCC, the compiler, could have helped us perhaps detect some of these things. So in the past, we begin seeing, like the G-Lib C developers, we begin shifting towards prevention. I'd say probably one of the most interesting things, and I'm a member of the Upstream G-Lib C Development Project, is we see a shift to WER, which is a real tight coupling with the compiler to make sure that if the compiler detects something, we're really going to go look at what the warnings the compiler has provided. And we're going to work with the compiler upstream to make sure that in that closed loop, the compiler is providing us good information about what it is that we're compiling in the sources, and that we're either correcting or adjusting those false positives. So this is looking back at G-Lib C, saying, hey, we've got these CBEs, we're going to shift towards prevention. And we begin to think that by W error, having the compiler always generate errors when we would have might have had warning. Now, what about the compiler side? What was the compiler doing at this time? So, you know, 2012, 2013, we begin significant compiler improvements, you know, in warnings infrastructure, in static analysis infrastructure, and that's based on common industry feedback. And so we begin developing the framework in GCC for the future static analyzers that we're going to be using and also for the warnings frameworks that we're going to be using to produce better warnings. And again, those better warnings coupled with W error, coupled with compiling the core runtimes of W error, are going to begin to produce that iterative cycle to improve the quality of the development process and the runtimes. On top of that, we have plugins and analyzers. And so one of the interesting pieces here is that, so G-Lib C has function pointers internally for certain operations that it does, those function pointers are relevant for us. Because those function pointers represent potential attack vectors. So in our case, enabling plugins would have been a plugin to look for function pointers in G-Lib C. And we'll talk a little bit about that in mitigation. So, you know, we see with the core runtimes, we've got CBEs, we move to prevention with the compiler. We're also beginning to shift towards prevention frameworks for detecting, improving warnings, plugins in order to analyze the code that you're actually compiling. But in 2017, we really, you know, we really turned towards improving the quality of, improving the quality aspect of prevention, which is this watermarking technology that we're going to talk about, which is Anibin and Anacek, which is actually a compiler plugin. And we'll get to that. So, you know, we look at the past and we say, we've had these CBEs in the runtime, the compiler, we really want to improve looking at the, you know, looking at the frameworks, looking at those things. And it kind of led us to these three foundational principles, which was obviously, I mean, not obvious, but, you know, when you look at them, you say, yeah, these are great. You know, prevention is better than mitigation. In many cases, if we can prevent or provide a higher quality FOS tool chain, then that means that downstream, customers won't have to schedule downtime. Customers won't have to review. Customers won't have to do upgrades because CBEs are present because we prevented them. But the important thing here is in the middle, we have prevention. We really want quality control to ensure that the compiler flags, the safety, the ABI, and the things that we want are all in place in the way that we expect them. And if either of those things don't work, mitigation is there if the prevention doesn't work. So let's talk about prevention in the compiler. So prevention in the compiler generally, two important things. We have buffer bounds checking and we have object size detection. Buffer bounds checking has been continuously improving since GCC9, array bounds, string operation, overflows. Those things are key because bounds checking provides us the ability to, you know, to look at whether or not you've overwritten those bounds. Object size detection is, again, important. And an important piece of object size detection is going to be marking up runtime headers with specific access attributes in order to allow the compiler when it's compiling a C or C++ application with those headers in that new markup to give you even better results when it's looking at warnings for compiling your code. So prevention in the compiler, we've got buffer bounds checking and object size detection. The, you know, the next thing in prevention in the compiler is the kind of dynamic aspect of the static analyzers which are, which had arrived in GCC10. The sanitizers, which are UbiSAN, T-SAN, A-SAN, they are shared with LBM and the new tool chains with GCC and Clang. And I would say plugins, plugins for the compiler. I think for G-Lib C, the interesting plugin was the plugin that Alder Hernandez wrote, which was a plugin, again, to find all those function pointers that could be attack vectors for the runtime. Because if you know, when you can modify a function pointer, you can modify the control flow of the program. And there are several ways, which we'll see in mitigation where we want to, we want to change that. So prevention in the compiler, we talked about prevention in the compiler, prevention in the runtime. The prevention in the runtime, the thing I want to highlight about prevention in the runtime here is annotations to library function headers. So as GCC's warning framework improved, it became apparent that one of the things we wanted to do is annotate headers in C and in C++ with additional aids to allow GCC to, for example, compute sizes of objects and also to know something about the parameters of those functions. And so we have access annotations now in G-Lib C that allow you to say, well, for this function, this parameter is read only, for example, or this parameter is write only. And so in that way, we've begun marking up the run times with access information. It actually allows the compiler to do a better job when compiling your code. That's an important thing to call out here as in for prevention in the runtime. So and I know I'm going quite quick. I do want to leave some time for people to ask questions and to answer those questions. But we've gone, we look at the past and we said, you know, we're shifting towards prevention. We said there are these foundational principles for the GNU tool chain here, which, which are, you know, prevent prevention is better than mitigation. We've got to have quality control. And at the end, we do need to have mitigation in the event that that your, you know, the prevention isn't enough. So what do we mean when we talk about quality control? So quality control, I'd say breaks down into three distinct things. The first thing I want to talk about is ABI tracking. Second thing is community process. And the third thing is the QE aspect of verifying that what you thought were your security mitigations or your security flags or security hardening that you put in place were actually there. So live Abigail is a, is a project whose entire purpose here is to serialize ABI artifacts, to be able to compare ABI artifacts, to be able to save from release to release. When I have a compiled library, C or C++, when I have a compiled object, what's its ABI? And then you can save from release to release. Has that ABI changed? That's all well and good. But the next part is upstream security process, which is you have to tell your users what assurances you provide about ABI. When does the ABI change? When does it not change? That's important because there's a lot of security defects that have to deal with sizes of things changing. And when those size of things changing, you get buffer overflows, you get reads of non-deterministic values. And so it's important that, that for an upstream project, you set security expectations appropriately. In G-Lipsy, we've been working on having a well-defined security process with exceptions. It helps the community understand where we will accept things and where we won't. So for example, some people often come to us and say, well, it's a regular expression engine, something that has security assurances. How much memory can a regular expression consume? The regular expression engine actually has a set of exceptions for it because theoretically you can write a regular expression that consumes a lot of memory. So you'll have to restrict the process if you're processing regular expressions and you want it to consume a finite amount of memory. But at this point, I would say we've done ABI, we've done that public security process with the community. The next piece we want to talk about is setting quality baselines. So within Fedora and within REL, we've been we've been working with a compiler plugin that is, that's called Anabin. Anabin has a dual, which is Anarchek. Anabin, as a compiler plugin, looks and records all of the options and operations the compiler took in a translation unit. It records them in the binary as metadata. So that at the output, when you're done producing all your binaries, you can run Anarchek and you can look over the analysis of what made it through the binary process. Did any functions attempt to disable source for incubation? Did any functions attempt to disable or change optimizations? And what this allows you to do is record what's there and then at the outset of the process, you can look to see, did I comply with whatever policies I had in place for the compilation of my binaries? For example, did I want fortify source equals to enabled and all my binaries was it ever disabled over some code? I will mention an interesting point here that there are parts, for example, of your runtimes that you may not be able to instrument. So again, you know, when you get to the Anarchek side, we do, there is this process of exceptions. But in general, once you have assured that you've got this API stability, once you're assured that you are, you know, your community understands your processes for exceptions, you then set that quality baseline for your for your generated binaries using using Anabin and Anarchek. So we've looked at, you know, prevention, we've looked at quality control and now we look at like what's the tool chain doing for mitigation for securing GCC and G-Lip C. In GCC, we've worked a lot on runtime traps. That's in in any case where you want to protect the runtime from carrying out unintended execution. You you you change that code. I'll give you an example here. F stack clash protection prevents jumping stack guards and it's generally done by the compiler instrumenting and probing the next page that you would access before you get there. The compiler mitigations are largely about runtime traps and runtime aborts to prevent bugs from being exploited. In the mitigation instance, when you detect that there's an exploiting process, you generally want to bring the process down to a state where it's failed safe. In in compiler and runtimes, we've recently added a new level of source fortification in which we tweak a little bit that balance between the runtime cost and the the amount of checking that we do. And this is a four to five source equals three. It was added to G-Lip C two thirty three and the implementation is immersion in LLVM and it's currently in progress for for GCC. And fortify source is that that again, we're looking at buffer bounds checking across across interfaces and across those interfaces to make sure that at runtime, the buffer bounds are what you expect. And, you know, it would be it would be remiss of me to not mention the belt and suspenders that we have with control flow integrity for hardware control flow integrity. So that's Intel CET and AR-64 is packed BTI. We have both of those in in GCC and in G-Lip C requires a compiler and a runtime and limit enablement aspect. And those are very interesting belt and suspenders when it comes to turning on for hardware that has these these control flow integrity state machines for tracking those state machines to make sure that the processes when compiled with CET are following the the are following what CET requires. That is to say that the control flow that is assured by CET is actually being met. This is very interesting for mitigating Rob cop and job kinds of tax. And so that we have that in GCC and G-Lip C. Now I had been mentioning in the past encrypting function pointers. The the the interesting thing about that is positive feedback that you get working closely with the compiler is you can develop plugins for finding function pointers in in G-Lip C and that you can come back and you can obfuscate those function pointers. And that's useful. You can obfuscate them with a bit of randomness. That's useful because in those cases we reduce ASLR leaks and and control flow exploits. So looking into the future, we have, you know, lots of project plans are focused, I think, remains still on prevention. There's key projects to improve mitigation. But let's let's dive into that. So prevention in the future, we've got brand new static analyzers in GCC for preventing use after free and double and double free. Those are going to be very interesting to watch them evolve to work with the compiler developers as we compile G-Lip C and compile the core run times and and and see how we improve those. I'd say future of G-Lip C. Something we've already done, which is removing malloc exploit primitives in the form of the current implementation was improved in G-Lip C-234. But I'd say there's there's almost this discussion of reducing attack surfaces, right? Removing the the unused converter plugins, dis allowing certain features, you know, from a from a from a prevention perspective. What we're doing is we're removing we're removing interfaces. So long as the user is capable of doing that. And containers are always useful because they allow us to play with some of these choices. You can have like a UBI minibus or you can have a minimal container that, for example, has a few of these things turned on in and provides different behavioral and also provides a different attack surface. Quality control, I think the interesting thing to mention about future of quality control is deploying sanitizers more extensive during OS development. I think there's this aspect of the core runtimes, particularly in G-Lip C, where we cut up the core runtimes and enable sanitizers for a larger portion of those runtimes, larger portion of the runtime components within G-Lip C. Some of the dynamic loader can't be easily instrumented with sanitizers because there's a chicken and egg problem. Sanitizers depend on the implementation provides certain functionality, so you can't always instrument that, but you can instrument a large part of G-Lip C. And I'd say mitigation in the future is going to be, I still think those sanitizers in the runtime. So in summary, I'd say prevention, quality control and mitigation are those key foundational aspects of what we're looking at. We looked at the past in G-Lip C and GCC, how that started, how our behaviors there started to trend us towards prevention. Once we thought about prevention, you're really into prevention, quality control, mitigation. What we've done today has been a lot of work in GCC 9, GCC 10, GCC 11, and in G-Lip C also to make sure that we have those pieces in place. And in the future, we've got lots more projects in order to ensure that the core system tool chain is providing you all the features that you need to harden and secure the applications that you're building with it. Thank you. And I am going to flip to check to see if I have any questions. So I'm just going to stop sharing right now. Yeah, you have a question already in the Q&A. I'll just read it. Sure. Can you speak to F-analyzer options added to detect CWEs and do you plan on enhancing the coverage for CWE detection? So tell us a word about F-analyzer. I don't have an answer for that today. So if you if you want to email me, we can discuss it. So I'd say there's probably a longer conversation to be had there. But thanks, John. And I'm going to see if there's a... How's this? You're not opposed to improving it. No, I'm not opposed to improving it. No, yeah. And I'm going to just look down. Sebastian Crain has a question. Do the principles you describe in the talk apply to the GNU Compilers other than CNC++ such as GCCGo and GDC? So the so Go is interesting and some of the principles we have here apply so long as the resulting output is an ELF binary and so long as you have, for example, dwarf debug information, because a lot of the way, for example, live Abigail and have been in Anachec, a lot of a lot of that relies on us having dwarf debug information generated by the compiler that we can use to reason about types and interfaces. It relies on being able to have an output binary that allows you to store metadata in it. So I think these principles are all at a high level that are applicable. The primary focus for Platform Tools Team has been CNC++. There's a lot of code that's being built in CNC++. Yes, today we are like Platform Tools. Absolutely. We've got teams working on Go and on Rust. So there are very similar questions about, you know, how we extend these things. I think that's very much a work in progress. Excellent. And let me go back. All right. Anyone else have any other questions? No, I'm very happy to take questions by email. Also, if you want to reach out to me, so. Yeah, do you want to post your email in the chat? I mean, that's up to you, but so. Oh, absolutely. Absolutely. Well, not like you couldn't find my email just by Googling, but yeah. Right here, Carlos at redhat.com. So excellent. Yeah. And I I've tried really hard to stay on time there for you, David, so that we can keep everybody going forward and getting the rest of the talks in place and getting people back to their their daily schedule. Well, I greatly appreciate that. And I'm sure everybody else does too. I will note that in this particular case, we have the interesting thing that we have a break coming up. So, but I'm sure people could use it a break. So always, always. Absolutely. Right. So, so folks. So first of all, Carlos, thank you very, very much. I really appreciate the work you and everybody else is doing there and for you helping to share about it. All right. So thank you very much, everybody. We have we have a break scheduled up next. We are scheduled to return at twelve fifteen with lots more interesting talks to come. So please take a break now and we'll see you at twelve fifteen. Thank you so much.