 All right, how's everyone doing? Good? Energy good for an afternoon talk? Don't worry, this will be energetic. So we can wake everyone up, anyone who's slightly out. All right, okay good. So today's talk is gonna be, well, it's a very long title. Don't get owned by your dependencies and it's about how Firefox uses WebAssembly to protect itself from all of its exploitable libraries. And this work is actually like a multi-year collaboration between industry and academia. So folks from UC San Diego, UT Austin, where I'm an assistant professor, and Mozilla of course. And this has been like three plus years of work. And in a nutshell, what today's talk is going to be about is this idea of in-process sandboxing, which is powered by WebAssembly. And in a nutshell, what we're doing is we are sandboxing native code library dependencies. So you know that random C library that you pull out of GitHub and you need for your application that you can't audit but you still need to use. We want to sandbox that. We don't want to take on all of the bufferable flows and use after freeze of that random C library that we anyway need to use. So this is an old idea actually. So it's been introduced in academia 30 years ago at this point. So that's how long ago it was first introduced. And this is the first time it's actually making its way to production and sustained use in production for millions of users. And what made this possible is of course WebAssembly. We have this nice widely deployed production ready tool chain which is awesome and gives us sandboxing. And this other tool we had to write called RLBox which makes it practical to take those same libraries that you rely on in your application and put it in a WebAssembly sandbox and just have everything work. And some of that is easier said than done but that's why we needed to write this framework. It's really hard to do that by hand. And so this is something Firefox has been shipping for three plus years. The stuff I'm telling you it's of course in the context of Firefox but all of this is completely general. If you have a C++ application you can take RLBox and do this too, okay? So outline, this is briefly what we'll do today which is of high level motivation. Why we care about any of this stuff? We'll then talk about all the pain points of actually using WebAssembly as a sandbox for libraries and why we really need a framework to help us make all of this work out. And then the framework we wrote called RLBox is this framework so I'll talk a little bit about how it does what it does. And then we talk about our experiences deploying RLBox and hopefully this is particularly of interest which is the timeline it took for us to actually get this working in production with millions of users and so on and all of the stuff that we did on the way. So without further ado, let's just jump right in. So the reason we care about any of the stuff we are talking about now is almost every application we use uses a ton of C and C++ code which is fine. It's great, it's fast, that's what we want but this sort of native code also has a ton of memory safety issues. We know this, 70% of our bugs in large applications like browsers and operating systems tend to be memory safety bugs and these are studies conducted over five years, 10 years, 15 years, it's more or less always the same stat, right? And okay, so you might think, well, we have Rust now. Why is any of this an issue? Like just forget it, let's just do Rust. Yeah, here's the thing. Safe languages like Rust use the same libraries under the covers and if you don't believe me just go to Cargo's website and just search for CC. Everything that uses the CC create uses the C compiler meaning it has some C code in its compilation that it relies on. So you're ending up using the same C libraries and you are not as safe as you would like to believe. And the issue really with including all this native code is that native code library bugs, they're used in real in the wild attacks not just like these theoretical things that you might see real in the wild attacks that have targeted iMessage that have targeted browsers, a ton of software. In fact, there's like the actual list is endless. I can't put it on one slide. There's just like a few examples and very broadly our defenses that we have against all these things don't really work. Like we have things like ASLR and stack canaries and CFI as all mitigations and all of these definitely introduce speed bumps for attackers, they do. But they are ultimately routinely bypassed. It's just part of the process. And then, I mean, you could, again, we could try to rewrite all of our code from C to Rust and we should totally do this when we can. But this is not just the simple matter of taking your C syntax and moving it to Rust syntax. You have to redesign your library quite often or redesign your code. You have to test it for performance. You need feature parity. You need to make sure there's no regression tests. It is not an overnight process and you are billions of lines of existing C code that is out there is not gonna go away anytime soon. So the reality is we are gonna have to live with some C code and we need to do something to protect it. So the most straightforward thing I can think of is, well, let's take all that C code and let's just push it into a separate process. That way a memory, a buffer flow in that separate process won't affect my main process. I'm good. And this does work to some extent in some settings. So a lot of caveats. And the main reason I put these caveats on is when you use a process to isolate your C code, it means you need inter-process communication every time you wanna do anything with that C code. So every function call is now an inter-process communication. So it's very, very expensive. Also, this sort of re-architecture of your application, though it sounds simple on paper, it is very hard to do in practice because you would be fighting all of these performance costs all the time. So the engineering costs are really high too. And I'll just read this quote from Chris Prama from Google Chrome Security who has an entire talk on this one subject, which is real-world operating systems put a ceiling on the effectiveness and applicability of sandboxing with processes. And I think that's a great quote. It summarizes the state of where we are today. Browsers have pushed the limit of, for example, how many processes you can spin up. We can't keep spinning up more and more and more. So what we instead are going to do is in-process sandboxing. And in particular, in-process sandboxing with WebAssembly. Because this is what WebAssembly is good at. So what we want to do with Firefox is rather than build it in a single monolith, like one big binary blob, we want an architecture that looks something like that. Which is different components are isolated. The code and data are isolated. So for example, libJPEG is isolated. I say, let's say we put it in this memory range of OX4100 to OX4200. And it's isolated from everything else that's running on the system. And the purpose of this, of course, is if you have a buffer overflow in libJPEG, it's not going to affect Firefox. It's going to be contained. And the advantages of re-architecting Firefox like this is, of course, we get all of the security that we want from the isolation. But we also do not pay for inter-process communication. We are not paying those crazy performance overheads of spinning up new processes for each library we want to isolate. The disadvantage is, this sounds simple to do. Engineering-wise, we still haven't solved the problem. There is a lot of actual low-level plumbing we need to do to get all this actually working. And this kind of segues nicely into the next part of this talk, which is, why do we need automation or some sort of framework to help us with this? Why can't we do this by hand what's so difficult? And the easiest way I can show you what's so difficult about this process is by walking through an example. So for the next few minutes, all of us are Firefox developers. And we are tasked with this job, isolate LibJPEG from the rest of Firefox, get Firefox into this architecture. OK, step one, very simple. I'm going to take LibJPEG source, compile it through my favorite Wasm compiler, and produce an isolated binary. So this part is standard. Off the shelf, compiler, all of this is great. It just works. Step two, and I mean, this is a WebAssembly conference. Everyone is familiar. But in case you're not, the way WebAssembly does its magic, and this is a very simple approximation, is just bounce checks. Every time you have a memory access in WebAssembly, it is checked to belong to this range of 0x4100 to 0x4200, for example, and therefore you can never have an out of bounce. That's how it does its isolation. Now, there's a bunch of optimization, bunch of tricks to get rid of those bounce checks, because they are slow, but that's the intuition. Now, back to our modification of Firefox. We've compiled LibJPEG to WebAssembly. Step two, now we just have to modify Firefox to use this LibJPEG. And this really does involve a lot of steps more than you might imagine, because when you use this LibJPEG, it's just not about, you know, call the right functions. You have to do a bunch of stuff. The first thing you have to do is really decouple the library from the rest of Firefox. Normally, when you use a library in an application, things are intertwined. Code is intertwined. You have functions calling, other functions, calling back into third functions and so on. There's all sorts of code intertwining and data intertwining. You have shared data structures that go back and forth, shared globals, all of these things. And you have to think about these things. So, first we have to decouple everything fully, and then we kind of have to go back and think carefully about what we share, because you can't just say, well, these things are separate and you're done, because then there's no communication. You do have to have some level of sharing, but just enough to make things work. So you have to figure out all of those things. And this is made challenging by a very low level problem, which is almost at a different abstraction level, which kind of throws everything off, which is WebAssembly has its own ABI, application binary interface. Meaning, when you have C code with other C code, you can just call it, right? If you have C code and WebAssembly binaries, you can't just call it straight. There's usually some routine of like, well, the first parameters, foo, and all functions begin with bar, and there's something happening with the binary interface that we need to resolve. And it's super complicated, because WebAssembly and regular host code can't even agree on a pointer size. WebAssembly says pointers are 32 bits in 32-bit memory, and the host might be running on a 64-bit system and says, well, it's 64 bits. So even data structure sizes won't agree now. So this is actually more complicated than it looks. And additionally, let's say we solve this problem. Now we want to share stuff. We have to marshal things back and forth. If we marshal the world, like marshal or serialize the world, and pass it back and forth, you've introduced a ton of slowdowns. That's gonna get you nowhere. So you have to do this lazily, you have to do this very carefully, limitedly, and so on. And finally, and most importantly, you can't trust anything coming out of this isolated libjpeg. The point is it's untrustworthy. So you need to go and figure out where you need to add sanitization checks. So all of the outputs should be sanitized. All of the control flows should be restricted so that when it calls back into the application, it does something sensible and can't just call random things. So these are the broadly the three challenges. And to really drive this home, I'm gonna show you code. Don't panic. We don't have to actually read all of that, but just take my word for it that this is a super simple example of using libjpeg. The few things in reddish brown are function calls into libjpeg. And this is what a simple jpeg decoder might look like. Now if I were to go in by hand and do all those three things we talked about on the previous slide, which is decouple, marshal, add serialization checks, fix the ABI, et cetera, the code's gonna look like this. And this is, again, don't bother reading that. That's gonna give you a headache. But the point is this is a 10 line example and I've changed nearly every line of code. This is unsustainable, especially given that the real jpeg decoder in Firefox is a few thousand lines long. So this is not, I mean, just imagine the poor teammate who has to code review this. This is not gonna work, right? And even this slide is hard. Think about this in practice. Real systems are huge, thousands of lines. And the real issue with this is that the low level details of sandboxing, all of the IABI stuff, all of this security stuff is all exposed and the resulting code is a mess of features and security. So it's unmaintainable and it's difficult to update, it's difficult to test, difficult to debug. And imagine now doing this for all of the libraries you want to sandbox because this is not a one-time effort, right? Like you have to do this for each library you care about. So this motivates the next thing which I'm gonna talk about which is the existence of this framework called RLBox, something we've built to really simplify this process. And from a 50,000 foot view, what RLBox is, is it's this framework to retrofit sandboxing and it uses types, types as in the programming language sense to make sandboxing compositional and something that actually works for this case of library sandboxing. So it's gonna use types to hide the low level sandboxing details. You don't have to worry about all these IABI conversions and all of these things. It's gonna even take care of the marshalling for you. It uses types to track untrusted data and control flow coming from this isolated web assembly. So in short, if you miss a security check, it's gonna give you a compile time error. And we'll talk a little bit more in the concrete example in a second. And finally, types are actually going to let you make changes. So all those changes you saw, you can make those changes incrementally rather than all at once, otherwise your application doesn't work. In fact, RLBox lets you literally change one line at a time, call it quits and ship your application to production and come back to the next day to change the next line and do all your changes one line at a time. And everything still works, all your tests still pass, you can actually ship it to production. And all of this pure C++. We're not gonna give you, you're not gonna change your compilers, you're not gonna change anything else. Everything just works in your build system. It's good. So concretely, what RLBox does, it has only three different rules broadly. One is your function calls. Instead of calling them directly, you're gonna call them through this RLBox API called sandbox.invoke. And you're gonna, basically it's the same, but and if you forget, again, at any point if you forget to make any change like this, which RLBox requires, you're gonna get a compile time error. Next, any data coming from the sandbox has to be marked tainted at the type level. So for example, if you're reading the size of the JPEG image, it's not a UN32, it's a tainted UN32. Again, if you forget to make this syntactic change, you will get a compiler error. Finally, the rule is whenever you use tainted data and you do something like a mem copy with it, you're actually using the data, you need to verify that tainted data. If you forget to verify, compile our error. But again, if you wanna just pause, call it quits here, you don't wanna think about the verification process, you can just ignore the tainting with unsafe unverified. And this is what I mean when I say you can pause. You can just pause things here, ship this, it's totally fine, it's not fully secure, but it absolutely works and it's no less secure than what it was before. And so let's go back to that code that I just showed you and do these three things. And again, I'm just gonna show you the diff and I want you to broadly just stare the amount of changes now that are happening. So hopefully you get a sense that this looks slightly cleaner, slightly more readable, you can still see the original source there. And that's really the goal, which is this thing, our box is trying to be, is very minimal. And it's doing a bunch of automation behind the scenes. So for example, one of the things that's happening on this line, even though you can't see it, is there's a bunch of automatic ABI fixing because we are dereferencing a pointer to a data structure which has more pointers and so on. WebAssembly and the host don't agree on sizes, pointers are not the same. So there's a bunch of stuff like that that is happening behind the scenes. What's more, we are dereferencing a pointer without checking it, which could be an arbitrary pointer. So our box is gonna add in a bounce check there for you as well. Next up, oh yeah. So we've made all these changes. There's one more important thing to note. That size variable is now tainted because it's the JPEG size. We've read it, we've read some output from the WebAssembly sandbox. It's now tainted. And we're using it in a mem copy in the next line. So we basically have to verify this or sanitize. And this is the manual step that has to happen with you as a developer, which is you need to think about what the sanitization is. And here, the sanitization is simple. I just want the value to be something smaller than my output buffer size. And hopefully that makes sense. Like, is everyone following, tracking? Okay. And that part is manual. And there's nothing on the earth we can do to make sure that you've got that right. You could have totally said, no, no, no, I'm good with anything. And then you're not secure. So that is a part of the system that is manual. But you'll note that it's exactly one small piece of the changes. And what this really lets you do is you've now identified all of these pieces of code that you need to keep an eye on, audit, have your security people look at. And that's the only part you really need to pay attention to. Everything else is fine. It has to be, by definition, by construction. And so what we're trying to, the point we're trying to make out of all this is with, if you use WebAssembly and RLBox in this way to isolate your libraries, you do actually get this combination you're looking for, which is WebAssembly gives you the fast sandboxing and RLBox will give you the low engineering effort. So let's actually see what this looked like in practice, right? Like what does this look like to use in a real application versus a 10 line example? So the first thing we did is our tests, internal tests, which is how do we, let's deploy this across a bunch of applications and see if the application still works. This is all on my local computer. This is not deployed on all these applications, actually. But we took a bunch of libraries, tested them, took out, saw how long it took, measured performance, and so on. And broadly what we saw was the amount of automation RLBox was doing was like dozens to hundreds of security checks that you otherwise would have to do by hand, all of this is automatic. The remaining data validation like you saw is about like two to four lines on average. And all of this takes a few days to a week of sandboxing effort for a library, on average for a developer. So okay, what about in actual practice? Okay, let's look at Firefox and this is something we actually shipped, which is what does it take to sandbox XML parsing and font decompression. And these are the numbers we saw in terms of overall application overhead. And as long as these components are not necessarily on the critical path of your application, these are totally overheads we can reasonably allow. And it gives us a lot of security because once we find, once any bug is reported in one of these libraries, it's totally fine to say, look RLBox is there, it's gonna isolate this, we don't have to run around with our hair on fire, deploying a patch overnight and having this hit a million users, tens of millions of users instantly. It's totally fine, it is isolated. And hopefully this is interesting, I also put together this little timeline because one of the things we also realized during all of our efforts is that we spend a lot of time on tooling. There are two pieces of work that we've been working, spending our time on, which is one is of course, sandboxing these libraries in Firefox. But between all of that, we spend a bunch of time building the tools to make all of this happen and make our lives easier. So to start, we of course built our initial version of RLBox and we picked up this compiler, web assembly compiler called Luset, which was maintained by the Fastly folks and we forked it, optimized it for library sandboxing, made a ton of performance improvements for this case and we actually tried to deploy this. And since this is something hitting production for the first time, we also want to be cautious. So we don't target everything, we target just Mac and Linux and we sandbox the single library. And we let this run for over here just to make sure all of this works. This doesn't crash a bunch of machines, we don't get a billion people complaining or a million people complaining, oh, you just broke Firefox and so on. So that's how we started. But in the interim, we actually started moving from, we started doing a bunch of things. The biggest thing is we moved to this other wasm compiler called wasm2c. Again, we forked it because typically what we see is WebAssembly compilers are not optimized for library sandboxing, they work but they're not as fast. And we have a bunch of optimizations that we contribute because we spend a bunch of time. We added Windows support, we added a bunch of like sanitizers, things like, I mean, there's a very cheap memory sanitizer kind of thing that we built and so on. But the advantage of wasm2c is it has this compilation process that makes it compatible for any platform. So what we did next was actually deploy it to every platform. So Windows included, and this includes Android too. So it's on your mobile phone, which is something which is very rare for technology security frameworks in particular because it's very hard and expensive to get things on your phone because the phones are so much more resource constrained than desktops. And this time we sandboxed the whole bunch of libraries. And all of this is deployed and running and basically there are no issues we see in production. And occasionally we do see things. So we also spend a bunch of time in our application hooking up all of the telemetry, crash reporting, how do you optimize the sandbox creation? So one is already ready when you are looking for it. So there's like a pooling process going on. And we also switched back to, away from our fork and contributed a bunch of our performance optimizations to wasm2c, the upstream one. So shout out to the wasm2c folks who actually worked with us to make all this happen. So wasm2c is probably one of the fastest web assembly compilers today because all of a bunch of our performance optimizations landed faster than wasm time if you're curious. So I'll talk a little bit more about that in a second. So where we are today is we are deploying, we are of course sandboxing more libraries. One of them I'll call out is this audio speed adjusting library. And the neat bit is that requires simd. So we are actually deploying this with wasm simd support now, which again is a pretty large test case. 20 million users using wasm simd in the setting. It's a pretty good test case and pretty good stress test, especially for folks who are looking at what the performance of wasm simd is compared to native simd, whether you want relaxed wasm simd, all of these things, right? And going forward again, we're going to spend a bunch of time on tools, bunch of time doing research. We're trying to work on a version two of RL box which makes it even simpler. And I'll talk about some of the features where improving wasm to see performance even more using like stuff that we've just pushed out in research papers and various research venues. So I'll talk about that in just a second. And actually Tal has a talk about some of this stuff tomorrow, so I would recommend you check out his talk. And we're also working on doing a bunch of stuff in WebAssembly security. So, and let me briefly tell you about some of this stuff. And some of this is a mix of research and deployed research that's now essentially production ready, right? So I'll call out where each of these things are. So on the performance front, and our group broadly has been doing a bunch of work, including things like optimizing and implementing WebAssembly simd and WebAssembly's threading support in wasm to see compiler. We've also have this research paper out with this optimization called Segway which builds on x86 segmentation and optimizes WebAssembly. And the neat thing is with complete backward compatibility with just the code generation difference, we can speed up WebAssembly by about 7%, or sorry, eliminate about 7% of the overhead. Yeah, I said that right. And so that's pretty significant because eliminating the 7% or 7% out of the 30% or something of overhead is a pretty large chunk of it. We also have ongoing work and this is published research with Intel about this extension called HFI, again covered in Telstalk tomorrow, which is what is hardware support for something like WebAssembly gonna look like on CPUs that we have today. So we want backward compatible hardware extensions which work with complex CPUs that we already have today, not Greenfield. And we work with Intel folks to make sure this design is sensible and ready for some adoption for an x86 platform. We're talking to the RISC-5 folks to make sure it's ready there. So a bunch of work is going on there. On the kind of memory optimization thing and we are looking at a new version of RLbox, RLbox V2 and one of the core features which RLbox V2 will have is this idea of migratable heaps which is WebAssembly has this cool property. You have a heap here, you can totally move it over there and things continue to work. This is much harder to get working in an application setting because if a library says I have a heap here, I have some pointers here and the application says, okay, now I'm gonna create some pointers to that thing and the library just moves its memory. That is not so easy for the application to deal with. You kind of have to really think about how you change the code to make all this work out. And we have an idea for this. So this is something we're working on in RLbox V2 and the neat bit about this is we don't have to pre-allocate all our memory ahead of time. So this reduces the footprint that WebAssembly has in a setting like Firefox. On the security front, we have this tool called Verivasm which basically is this verifier you can run on binaries produced by WebAssembly. And what it does is it looks for the security checks that should be there. If there's a missing security check, it'll yell. So it's there to find compiler bugs and we ran it on a bunch of previous CVs that exist in wasm time and previous compilers and it's able to catch it. And it's there to make sure like when you have a powerful WebAssembly compiler that's optimizing everything, you don't optimize something you shouldn't. And it's actually been adopted by the Cranelift tool chain. You can totally run it today. So we also have Wave which is a verified WASI implementation. If you wanna talk about any of these things, feel free to find me and I'll chat more about this. We also have this tool called Swivel which is a WebAssembly compiler that builds in Spectre's hardening. So the full Spectre hardening is actually quite expensive but we've narrowed down a very small subset of all the hardening we did which will give us 90% of the hardening for about 1% performance overhead which is a very nice trade-off in my opinion. So I think that's something we're working on getting to an RFC style so that we can actually get broader adoption in various compilers. And finally, on the usability front, again, this is something we're building into RLbox V2. I showed you some of the examples of using RLbox. This can still be made simpler. We can still require less boilerplate and less ceremony. We can integrate more auto-martialing and sandboxing paradigms and C++ memory management paradigms, unique pointer, all that stuff and make all of that work. We can also support C++ libraries, not just C libraries. So all of these things are things that we are trying to get into RLbox V2 including, you know, and we're also trying to get like the basic stuff from WebAssembly like debugging support and stuff onto these production-ready compiler tool chains like Wasm to C. So I'll quickly summarize. Oh, just before I do that, you know, try it out. It's all there, it's all open source. Firefox even has a bug bounty if you're in the mood to kind of attack WebAssembly and attack RLbox. There's a bug bounty for you. So in fact, you get to start with a memory corruption. Put a buffer overflow anywhere you want in one of the libraries that we've sandboxed and break out of the sandbox. There you go, bug bounty. All right, so very quick summary. This new idea that we are exploring is in-process sandboxing powered by WebAssembly and we want to isolate all of our libraries that we feel are buggy that we are application nevertheless needs. It is an idea that's actually 30 years old and it's been in production for the first time since then for such, especially such extended use. And the key to making all of this practical was of course WebAssembly and RLbox to make sure all of this integration actually flows through. And it's been shipping in Firefox for over three years. So if you're interested in stuff like this or interested in deploying RLbox in your application, come talk to me. There are contacts and yeah, thank you for your attention. Happy to take any questions, going once, twice. All right, so come find me in the hallway track if you're interested in chatting more. Thanks all.