 We have Julio Moreno here. He's going to talk about optimizing sandbox creation with a FUS file system. Is that right? That's about right. Give him a warm welcome. Thank you. Is the timer working? Great. Hello, everyone. As I've been introduced, I'm Julio Moreno. I work for Google. I'm in the Bazel team. It's my first time I've fought them. So I'm going to talk now about Bazel. So today, I want to talk to you about Bazel and specifically about sandboxing and how we've been trying to optimize it to be faster by using a FUS file system. We have 15 minutes, as you know. I'll fill them up with my talk. If you have questions, we have to wait until later. But you can find me. I'll let you know how. OK. So before we get into some boxing, I want to recap a little bit what Bazel is. Even here, either last year or the year before, our team had a booth. And I know it was very popular. But this year, we don't have one. So if you don't know what Bazel is, go to the website at the bottom, Bazel.build. Or basically, I want to tell you, Bazel is Google's build system. And Bazel itself is the external version of it, the open source version, which essentially lets you build and test any kind of project that you have. And it's specialized in integrating trees, source trees that have many different languages. And the goal is to build anything very quickly and reliably. And reliably, I mean, you want your builds to be deterministic. So if you build the same thing twice in a row, they should give you the same results. And that's actually where some boxing comes into play. But before we get into the some boxing, we have to go ahead and tell you a little bit how Bazel actually models things. So the basic concept we have to understand for this talk are Bazel actions. Bazel action is essentially a command invocation. If you're familiar with any other build tool, like, for example, make, any command that make runs essentially becomes an action in Bazel. And Bazel represents this in memory with a data structure called action, of course. And the action contains a command line. In this case, we have an example for a CC compile that takes a source file and generates an object file. And as part of the action, we register in memory what the inputs of that action are and what the outputs that we expect from it will be. Now, you will notice that the inputs here contain, like, the compiler itself. I've simplified it by just listing the binary. But, of course, that includes any libraries that we may depend on, et cetera. But the important thing is to see that we have the source file, parser.c, as well as any includes that the source file might have inside, right? A c file, we have, in this case, the parser.c file has include parser.h. So that becomes part of the inputs of the action. And then when we run this command, we expect that the compiler will generate just one single.o file in the same directory where we run the command. Now, this is great. It works. But now the problem is, look at that dash capital I dot, right? The c compiler, when we, that was our memory structure, when we put the file system into play, the file system has more things. And in this case, in the same directory, we have the parser file, the header. We have another header called lexer.h, right? There is nothing preventing the compiler from reading that file, right? If your parser.c source file contains an inclusion of this other header file, directly or indirectly, and you haven't told Bazel about this, right? It's not part of the memory data structure. Then things will not work, eventually, because if that header that you didn't know about changes, then Bazel doesn't know that it has to rebuild this action. And then your build will not be correct in the end. So we want to prevent these kind of things. And the way we do this is with some boxing. Now, with some boxing, we have two things that we have to take into account. And the first thing is actually isolating the process. So when we run the compiler, it can only do the things that we think it should do, right? So here we have our process. Now it's a more clever version of cc. And we put it inside a sandbox. Now this sandbox will prevent things like, you know, it happens that this compiler wants to check the whole thing with the machine. Or for some reason, it wants to access the internet. Or it wants to access the file that we didn't declare in our inputs, right? So the sandbox will block all those accesses or mock the result and make sure that the process behaves only in the way that we thought it should. On Linux, we implement this today with username spaces. And on Mac OS, we use this deprecated tool called sandboxexec, and you have a couple of links at the bottom that explain all of this in more detail. Now, OK, that's how we actually prevent the process from doing things. But I'm not going to touch any more of this in this talk. What we want to look into is how we actually prepare the file system for this to work, right? Because we have to run this command somewhere. And the way we do this is we create kind of a truth environment for the command. So essentially, we have the same data structure as before. But now, when we want to run this command, right, the CC binary, instead of running it in the source tree, we create a separate sandbox directory that contains only the things that we want the compiler to have access to and see. And then we create this sandbox before we run the action. We run whatever it is inside there. We don't use truth, but essentially, it's the same idea. We just execute the command in that directory. And then we extract the outputs that we generated in that directory and put them back where they belong. In this example, you can see that they go into the workspace. That's not exactly true. They go in a different location, but you get the idea. Now, the problem is that the sandbox directory today is created with SimLinks. All of these things in between are SimLinks that point back to the workspace or wherever the outputs are. So what I put there, like, read only question mark is, like, we would like those files to be read only, but with SimLinks, we cannot do that. Anyway, the main problem here that we have and the performance issue that I want to talk to you about is that there are, in big builds, actions tend to have thousands of inputs, right? So then this process of creating the sandbox for every action becomes extremely costly. We have to do one SimLinks system call for every input. And when you have any kind of perturbance in the timing for the action, the action is in the critical path, so any increasing performance there will result in a big impact on the whole build time. So we want to minimize that. So the idea here is we will use our FuseFile system to make, to actually replace all those system calls with just one kind of RPC, right? We introduce a process in between Bazel and the file system that's called sandbox.FS. And this process receives calls from Bazel that tell it what to do. So in this case, we want to run the same action as before, right, with different files. I don't know why, but we have an RPC called create sandbox that says, please create a sandbox for action one. And I want to put these files in. And I actually want the root directory of this action to point to a specific location that will be writable. And the source files have to be put inside in read only mode. These are not symlinks. These are just actual real files that will be put in the directory. So then some boxFS comes in, does some in-memory operations only based on this data, and exposes you those files in the file system. So we can then, with that, go into there and run the command. If you're familiar with Linux or Unix or whatever, you may find this very similar to mount that's just bind on Linux or null file systems on BSD. And that's essentially the same thing, right? We have implemented this. They let you do this. They let you do a bind mount for multiple things into the same location, not with just one source. The other main thing that this does, which is different than bind mounts, is that we can have a second action coming in. And we'll have to do the same process. But look there, we didn't have to remount the file system to apply those changes, right? We just sent another RPC to this demon that's running, and it just added more files into the sandbox with different permissions, different paths, whatever. And we didn't have to remount it. So the performance there can be better. It can be. It's not yet, but we'll see into that now. So how does this behave? Well, so I ran some measurements a year and a half ago on macOS. And this is mostly about macOS because that's where we had the most performance problems on Linux. This is pretty good. And we got these numbers. I don't have more recent numbers because I've been having trouble getting things to work properly with more modern builds, but this is just a proof of concept for now. So here we just have three different builds. Two of them are building Bazel with itself. And another one is building one of our pretty large iOS apps. And there you have the total build time when we run it without sandboxing. Now when we enable the simulink sandbox, the original one, we got these timings. So for Bazel itself, we see a tiny increase, which is expected because any kind of sandboxing will have some overhead. But then for the iOS app, the increase is massive. That's just not acceptable. We cannot have this cost. When you want to do interactive development, you cannot have this cost because people will not enable this feature. And actually, they'll be disabling it because of this. So with sandbox the first, we've got, for the moment, to this. The overheads for Bazel itself remain. But for iOS apps, which happen to have actions that are gigantic, they have many, many, many inputs, we've gotten the cost to only 50% for now. I'm sure that can be cut down much more, but at the moment, that's what we get. All right. So something else I found that went in this project is that it was originally written in Go, and I went through every writing rest just because, basically. But I want to tell you a little bit of what we found there or what I found. So first, we started with Go. I had an intern that came to write this project, and he did a very good job. And it was working by the end of his internship. We found that VS Code, for example, has very good support for Go. It was very nice. It didn't know Go, so just having code conditions that was very useful to get into the language. But then at some point, we hit some scalability issues. The Go runtime didn't like the way the Fuse libraries for Go work, and it was not behaving properly. We were hitting some very significant performance issues. And the code started to become pretty hard to maintain. That's my own critique of Go, I would say. There is no way or accepted way of adding annotations in your source code, like assertions or, I don't know, thread annotations. So things had to have a lot of comments saying how the code was supposed to behave, but the compiler cannot enforce anything. So at some point, I just wanted to learn Rust. And this is my set project, so let's learn Rust by rewriting this thing. So I did that. The rewrite was very difficult. I mean, learning Rust and getting up to speed with it is hard, but I think it pays off. Something specifically that I found is that VS Code has also support. And I kind of like VS Code for the reason I mentioned before, to learn the language. But for Rust, it's very slow. Compile times are slow, as you may know. And they get in the way, even for like tiny edits, to get the red squiggles under the code. It takes a while, so that's annoying. On the other hand, the code that we have today is much more sane, and I feel much more confident that it's doing the right thing. We're in the past. I had to look it and maybe trust it. But more interestingly, and the thing that kind of shocked me in the process is that, so as part of the rewrite, I was trying to copy the same logic that we had from Go into Rust to avoid having to change anything, make sure that everything remained the same. But in doing this, Rust didn't let me write those same ideas in the same way. The compiler just refused that kind of code. And it turned out that the old code had many threading bugs that were not visible in Go, or actually running the tests that we have. But the Rust compiler would just catch them and not let me specify the kind of buggy code. I kind of wrote down my experience with the red in that post there. You can take a look at one later. And some common issues, or common things about this process, I would just mention that PIPRO, for example, is a profiling tool also from Google. It integrates extremely well with Go. It's super easy to use. It was very useful in finding the performance issues. It works also for Rust binaries with some more effort. It's also very useful in that case. The main problem is that the Fuse bindings for both Go and Rust are not first class, right? Fuse is a C project. And the Fuse bindings that have been written for these other languages are kind of like a written from scratch. I wouldn't say they are very actively maintained. They are missing some features. Then you file bugs out, performance will get fixed. So that's a very big problem for where we are at. I don't know what the solution is really, except, yeah, we'll see. Other things that I would like to do here, and I would say this is an open source project, Wing-Wing. I would like help if anyone is interested. Basically, the main problem today is that we have 50% cost in performance, but I'm pretty sure it can be brought down. And one of the problems today is that the protocol that we used to send data between Bazel and Sunbox FS is pretty inefficient. It's very chatty. It sends very big messages. We could just make that smaller. Another thing I would like to do personally is that I have this other tool called Package Comp, or Package Compiler, which builds any kind of software from package source, which is an FBSD package system in a Sunbox. In the past, I used like bind mounts. And it was very complicated to get them to work on macOS and blah, blah, blah. Actually, the original idea of Sunbox FS came from this project. I wanted to do Sunbox FS for this project, but then I had the time, and I was just lucky enough to sell it as we would use it for Bazel instead. So then I could do it at work as a 20% project. So that was good. So I would like to integrate it there. And other things we could look into is Microsoft has come up with their own way of sunboxing. They call it Build Excel. And instead of enforcing things, they actually let the code run as it was, and they sanitize what the code did. Like they audit, basically, they don't prevent. That's very interesting, can offer much better performance, but we have to look into it. Maybe we can add the same ideas. And finally, you should know that Fuse for Mac is kind of not open source anymore, and kernel extensions on Mac are going away at some point. So these two things are problematic for using Fuse on Mac, and it's unclear what's gonna happen. With that, I'll just leave you with a couple of links. Here you can go to the Bazel web page, the Sandbox FS project page, or just you can contact me below. You have any questions that will be here today and tomorrow, you can find me or not. It's very difficult to find someone. Just ping me if you want in Twitter, and then we can meet anywhere. With that, I'm done. Thank you. Okay, thank you for your talk. Thank you.