 Thomas, thank you very much for being here. Looking forward to listening to your talk. Thank you. Hello, thanks for staying so late. We're going to talk about Solana and its JIT compiler and a couple of lessons we learned from fuzzing it. First off, I'm Thomas Roth. I'm a security researcher. I lead something called Kraken Security Labs, which is the security research division of Kraken. Normally, we do things like Playdome on Bitcoin ATMs, hack hardware wallets, all that kind of stuff. But before we get started, there are a couple of people we need to say thank you to. So for example, the Solana team, who are pretty good sports about all of this. Then Secret Clubs Edison, this serves a lot of recognition, as you will see later. And also the AFL++ team, who were nice enough to help me out a couple of times. Now, before we get into the fuzzing part, let's quickly talk about Solana. So Solana self-describes itself as a high performance blockchain, which means one of their main goals is to execute as many transactions per second as possible. If you look at, for example, Ethereum, you normally execute a couple of transactions per second. But on Solana, their goal is like thousands of transactions per second. Now, on top of Solana, you can run smart contracts. And if you've worked with, let's say, Ethereum before, you probably know that normally those contracts are written in domain-specific languages. So for example, on Ethereum, it would be Solidity and so on. But on Solana, instead, you can write your contracts in C, C++ and Rust and so on, which is pretty cool and it's pretty cool how they implemented it. And we are going to take quite a deep look at that. Solana is a proof-of-stake slash proof-of-history blockchain, and so no mining involved. Instead, we have so-called servers that are called validators that basically achieve the consensus on the network. And everyone can become a validator. It's basically just a beefy server, like 12 cores, 128 gigs of RAM. And anyone can basically participate in that. At any time in the Solana cluster, one of the validators is elected the leader. This is just a high-level description. The details are a bit more involved. But basically, at each time, we have one leader and that leader is basically in control of the order and the way the transactions are executed. And so basically, if we submit transactions to the Solana network, they would all arrive at the leader. And the leader would then basically decide on which order these get executed in. Then it would run the transactions, execute the smart contracts and so on, and then publish first of the order in which the transactions were executed. And then finally, a hash of the state of, let's say, the chain after they got executed. Then all the other validators basically take the order that was published, they execute the same transactions again, and they validate that indeed they get the same hash. Now, the leader position always rotates through. And so if we have, for example, a malicious leader, eventually it will get ejected. And so, for example, if our leader would publish a malicious state or, well, a fake state, then the validators could say, hey, we don't agree and basically eject the leader. That's just on the chain side, but what we are really interested in are the actual smart contracts. Now, Solana smart contracts can be written in C and C++, but most that you will actually see are written in Rust. Rust, I think, is a pretty good choice because you get all those nice safety features, you have memory safety, you have a lot of other things that you really want to have when your code is literally dealing with a lot of money. And so, it's a pretty good choice. Rust is a compiled language and so, as a compiled language, we also need a target. And in the case of Solana, they choose EBPF as the target format. And so instead of directly compiling to, let's say, X64 and run the smart contracts natively on the validators, they compile into EBPF. And so, a smart contract is literally, in this case, an EBPFL file just as you would find on Linux or whatever. And yeah, if you're not familiar with EBPF, EBPF stands for Extended Berkeley Packet Filter. Basically, it's used in the Linux kernel and also on like BSD for network filtering, for packet filtering, that's where the name comes from. And it's a pretty simple risk machine, like 64-bit risk machine, fixed length instructions. And in the Linux kernel, we actually have a full BPF VM. And so, if you do packet filtering in the Linux kernel, there's like a just-in-time compiler, program verifier, and it runs those BPF programs. The BPF machine is pretty simple. It's just 10 registers and a frame pointer. The instructions are 64-bits wide, and the encoding is super simple. You have one encoding that's just a default encoding for all instructions, and then you can also optionally have like, basically 128-bit wide instruction that way you have a 64-bit immediate value. We just have your regular instruction classes such as load and store and just some basic arithmetic. It's really not much magic, and as you will see, we don't really have to understand BPF too much to actually find vulnerabilities with it. And if you work in security, for me, I mainly know BPF from CVEs, and so I really know it as something that has a lot of vulnerabilities in general. And so, when I read that Solana uses EVPF, I got pretty curious how they do it securely, basically. Now, the EVPF ELF that you generate from your REST code is basically directly uploaded to the blockchain, so you can dump the program and you will just get the regular ELF file. There's no special custom encoding also, which is pretty nice because you can use all the existing tooling for ELF files on those smart contracts. You don't need all the domain-specific stuff, basically. Now, a couple of more things we need to know about Solana smart contracts is if you've worked with Solidity, so for example, if you've ever written an Ethereum smart contract, you probably know that Ethereum contracts store their own state. And so, if you have a global variable in Solidity, the value of that variable is stored as part of the contract, basically. And for example, here we have this public variable, just text, which has the state hello, and if we were to submit a transaction that calls the function foo, that state of the contract would get updated. This is not how Solana works. Solana contracts are completely stateless and they don't have their own storage, basically. The way you actually interact with data is that you have separate files, basically, that are called accounts, and these accounts basically pay rent to not get deleted. And so, it's pretty nice because if you download a smart contract, you have all the state you need because there is none, but you still need to figure out what files on the chain and so on does the contract actually interact with. Now, the L file itself, it's a shared object, and shared objects in the way Solana uses them have one big caveat. You can't have any writable global variables, simply because the shared object gets loaded once if you execute multiple transactions that run the same contract, you wanna make sure that the state of your in-memory shared object doesn't change and so we don't have any writable global variables. On the memory side, it's pretty simple. Basically, the elf just gets put at a fixed address, hex 1000, the stack starts at hex 2000, heap at 3000, and finally, we have something called program input at the hex 4000. The program input is basically just a byte array and if you execute a transaction that calls into the contract, the byte array is basically the argument towards the smart contract. So you don't call functions by name or so, instead you just provide a byte array and it gets somehow parsed by our elf file. Now, the elf file, once the contract gets executed, gets run by the Solana VM. And if you build a VM for, let's say, BPF, you have two choices. You can either write an interpreter for BPF or you write a just-in-time compiler. Now, an interpreter has a lot of advantages in terms of safety, because you don't generate native code, you can very easily sandbox it all, but it obviously has a performance impact. And as Solana cares a lot about high performance, it's very probable that they use a just-in-time compiler. And just-in-time compilers, historically, have been a security issue, let's say. They're pretty difficult to get right and that's just because of the concept of them. If you do just-in-time compilation, you basically take BPF code and you somehow generate native instructions. And if you're not very careful that somebody can't generate random instructions that they can freely choose, then you can very easily get into deep trouble because basically the compiler will just write into memory, then map that memory as executable and run it. And so if you're not very careful, your JIT code might just exploit your existing, like basically the process it's running in. You also need to bring your own memory safety. So let's say I have BPF code that accesses memory at, I don't know, address zero. I wanna make sure that this memory access doesn't actually happen on the, let's say physical process address zero and instead you wanna catch it and so on. And this is all a lot of work and it's fairly difficult to really do this securely because you just have a large list of things to do. And yeah, so this all sounds pretty interesting for some security research. And so we decided to look deeper into the virtual machine of Solana. You can find the code online on github.com slash SolanaLabs slash RBPF contains the full machine. It's written in Rust, which is great because we get memory safety and so on. And if we check the read me, we also can see that indeed they chose to use a JIT compiler that only at the moment supports 64 bit x86. But just because the virtual machine contains a JIT compiler, doesn't mean that it's actually used in the wild. And so we had to kind of dig through the code a bit and eventually just found that we should have just read the release notes because since last year in March, basically the JIT is enabled by default. For us, this is super interesting because it means everyone can upload smart contracts to the chain. If we manage to build a smart contract that can exploit the validators, we can do a lot of very malicious stuff. And so far, this was super interesting. Also in the read me, we can see that Solana RBPF is a fork of RBPF from Quentin Monet. And if you see that something is based on something else, it's always good to go to the original source and check issues, to-dos, CVEs, things they might have fixed. And in the case of RBPF, if we go to the GitHub repository, there are indeed a couple of interesting things. And so, for example, in the to-do list, there's literally a point, improve safety of JIT-compiled programs with runtime memory checks. Then you scroll further down and there's the section, what about safety? And it literally says, it will crash if your JIT-compiled program tries to perform unauthorized memory accesses. Now, crash is a very wide range. Like, crash can mean a lot. Like, crash could mean, you know, a rust panic, or crash could mean, oh, you can write into memory and, you know, it's sac falls. And also in the caveat section, there's a whole section called, the JIT compiler produces an unsafe program. Memory accesses are not tested at runtime, used with caution. Now, we could go and, you know, mentally check if they added safety checks around this, or we could do the lazy way and we just start fuzzing the VM and see what we find. And so, I'm lazy, so I started fuzzing. And fuzzing a VM is actually surprisingly simple, at least in this case. To fuzz, let's say, the Solana VM, all we need is some kind of input. So for example, a simple EBPF program. Then we start mutating it semi-randomly, so we just start flipping bits, adding bytes, removing bytes, and so on and so forth. And then we execute it in the Solana VM. Now, we wanna get this all as performant as possible. And so, the Solana VM, if we get lucky, we wanna execute this multiple thousand times per second. And so, we wanna try random mutations as fast as possible on as many machines in parallel as possible. And then, hopefully, we, you know, eventually crash the machine and we collect those crashes. And if we get very lucky, we can analyze them and we maybe find some nice vulnerabilities. That's the plan, and so, let's get started. Now, first, we need some simple EBPF programs. Now, if you wanna fuzz a VM, you wanna be sure that you get a very wide variety of inputs, you know, you wanna cover as many instructions as possible. In this case, we're looking at elf. Elf in itself is already pretty complex, and so, you wanna have a lot of different elf binaries, different types, and so on. And luckily, the RBPF code base contains test cases for a lot of things. And so, we can just take these tiny binaries that are already made to be almost the ideal fuzzing input, and we can just take them and feed them into our fuzzer, or into the VM. And now, one of the issues we encountered at this point is that we wanted to analyze these binaries. So, we wanna, before we put them into the fuzzer, we wanna look at the instructions that are generated for them. We wanna look at what do they do? Do we really get the instruction coverage that we require? Which puts us at a place where we wanna, how can we actually analyze them? At this point in time, there was simply no good reverse engineering tooling to actually look at the compiled output of a Solana smart contract. Now, there were, like, a disassembly that gives you a huge text stamp, but that doesn't really help you, because Solana contracts tend to be pretty complex and pretty large, and so, you can easily have a lot of libraries in there. You can have a lot of dependencies and so on, and it's very easy for these to get very large, and you really want to have an interactive disassembler to work with them. I'm a big fan of Ghidra, and unfortunately, Ghidra does not come with native eBPF support, but Nailin98 built a processor module for BPF, and so, with that, we can already, you know, Ghidra understands the machine code, we can do patching, we can do all kinds of things, but we can't load Solana binaries yet, because Solana has this custom memory map, we have all these custom things that we need for Solana, and so, we decided to write a completely custom loader based on this eBPF that basically sets up the memory map, finds the symbols if they are in the L file, and basically allows us to just drag and drop a Solana contract into Ghidra and analyze it. So, for example, if we have a simple hello world binary, we can just drag it into Ghidra, import it as Solana L file, hit okay, and then simply double click it, and to load it into the code browser, run the regular analysis as you always do in Ghidra, then you go grab a coffee because analysis takes forever, and then once it's finished, we can just, you know, navigate through the binary, we get this assembly, and we even get the compilation of the full contract. This is pretty nice because you can really, you know, dig through a contract. The decompilation will not help us with analyzing the fuzzing results because those, you wanna really look at the instructions, but this actually helps us a lot with something else that's unrelated to actual virtual machine security, but that is important for us to be able to trust public contracts because verifiable builds are pretty important. If you think about like a lot of smart contract projects, you get like a link to GitHub, and it's like the contract code is there, but you can't actually easily verify whether the contract code is identical to what's stored on chain. And so, for example, on Ethereum, if you go to EtherScan and you look at a contract, they have this thing called contract verification, where you can just paste in your contract code, they will compile it with the same Solidity version, and then they will tell you, yes, this is all verified. With Rust, it's not as easy. If you've done, you know, reproducible builds with Rust before, it's not very easy. And so, at the moment, most contracts on Solana, even the very big ones, are not verified. But with our Ghidra tooling, we kind of bring our own verification to the whole thing. So we can really just go, we can download, you know, any contract from the chain. So for example, using Solana program dump, you can literally just go and fetch the L file from the chain. So if we run file on this, you can see it's really just a regular 64-bit L file. And then we can drag it into Ghidra and hopefully, you know, analyze our contract. Now, obviously, we can't go and manually reverse engineer the entire contract because, you know, it's really, really large and really annoying. However, what we can do is we can use the Ghidra program comparison to actually compile, to actually compare our manual compilation with what we find on chain and get a very clear picture if, you know, the code is really identical. And so this really helps us because, you know, we sometimes need to trust contracts with quite a bit of money. So with that, we have our simple EVPF program. We have our analysis tools and we are basically almost good to go. Next, we're gonna first look at the, at how we execute our simple program in the Solana VM because we want to, those parts are running before we start mutating so that we can be sure that everything is running. And it turns out that this was really, really easy. So I said RBPF is written in Rust and so we can really just, you know, write a couple of lines of Rust to load our EVPF file into memory. Then we parse the executable. We did compile it. This is just using the RPF API. You don't have to read the code. You can get the slides and read it on there. I just want to show how short it actually is. And then we verify the executable. This step will become important later. We create the virtual machine and then we finally just go and run it. Simple as that. And so this is just like 20 lines of code and you have all the base that you need to start fuzzing the VM. So with that, we have our EPPF program. We have our VM test setup. Almost good to go. Next, we need to start mutating randomly. Now you could write your own fuzzer that's maybe even, you know, instruction set aware and so on and so forth. But there are people who are much better at writing fuzzers, especially the people at AFL++. And AFL++ is really nice because it has, you know, rust support. We don't have to write our own fuzzing infrastructure. It comes with native dual core and multi-core support and so on and so forth. If you are not familiar with AFL++, it's a fork of Google's AFL. It's a bit faster, has better instrumentation and it comes with rust support and the rust support is amazing. It's super simple to use and it's really, really fast. Also AFL++ instruments the target binary and so it's aware that as it goes through the JIT compiler which branches has it taken and it tries to automatically mutate the inputs to get as much coverage as possible. And so this allows us to really ensure that we get a wide range of the JIT compiler actually fast because as we are fuzzing it will see, oh, I haven't taken that branch yet and so on and so forth. And it also comes with a nice UI and it has features to minimize test cases. So if we find a crash and the crash is, the backtrace is basically identical to another crash, it will just ignore it and save you a ton of work. And what's really nice is that the rust integration for AFL++ is really just add a single macro to your code. Like we just remove the file loading and we just add the fuzzing macro. It gives us a byte array with whatever fuzzing input we have and then we are ready to go. Now because we are doing rust, we also wanna add a panic handler because otherwise any rust panic will be counted as a crash which we don't really care about because we wanna get memory corruption, not rust panics. And with that, we are already ready to go. We are basically ready to fuss with like 30 lines of code or so, pretty awesome. And so I was pretty excited about this because you just let this run for a bit and hope for the best. And so you just started with cargo AFL fuzz. You provide a directory with inputs which are just the example BPF programs. You provide an output directory with the, where basically the crashes and so on will be stored and then you finally go and launch the binary. So let's do that. And now we are fuzzing. And if you are not familiar with AFL, it comes with this nice CLI. It tells you on the top, basically the timing of your fuzzer, it tells you, okay, how long have I been running? What's the time since I last found a new path in the fuzzer, which is pretty important. It basically tells you, like if this goes very high, it means it just didn't find a new executable path in the binary and so you should probably stop fuzzing. It also tells you how many crashes has it found, how many hangs and so on. And it tells you the execution speed. And so in this case, we are running at like 3,600 executions or so per second which is pretty decent. If we let this run on like 12 cores for a couple of weeks, we get a lot, a lot, a lot of results. And so we had this running for a couple of hours and then something nice happened. We found a crash. And time to party, hopefully, we'll see. And we let this run for a couple of weeks or a couple of days actually at first and we got a lot of crashes. Unfortunately, if you look closer at the crashes, they tell you which signal caused the crash. Now, does anyone know what signal 08 is on Linux? Floating point exception. It doesn't really help us at all. It just means that we did something numeric wrong. No real luck. But however, all between the signal 08 are also three signal 11. And signal 11 is exactly what you want to find because signal 11 is a segmentation fault. And no matter what you do, a virtual machine should never cause a segmentation fault. So even if this is, if you see a segmentation fault, it's most probably a vulnerability. So I was super excited. We got three segmentation faults. Let's start looking at them. And so, oops, sorry, I started executing them and my target binary wouldn't crash. Basically, I tried it again, again, again. I wouldn't get a segmentation fault. I eventually wrote a script to simply run this like 10,000 times. And then it would only crash sometimes. What's going on? Well, it turns out that, you know, if I had read the AFL manually to the end and not just skip to the good parts, they have this indication for something called stability. And basically, stability tells you how deterministic your program is. Basically, if it executes the program with the same inputs twice and it sees that different branches are taken, it reduces stability and tells you like, hey, your binary is not deterministic. Something is sketchy. Like, for example, you might have an RNG that is doing weird stuff. You might have something else going on. But how could this not be deterministic? I mean, we are loading into VM. We are running the same binary. What's going on? Well, it turns out that RBPF has this feature that introduces random knobs into the code. And, you know, it's on by default. And so that causes, each time you run your program, it causes it to be slightly different in memory. And so obviously, sometimes it would crash. Sometimes it wouldn't crash and so on. And simply by setting this to zero, suddenly our stability goes up to 98. Yay. We actually can fast nicely now. And indeed, after running this for, I think, 14 days, we had 59 segmentation falls. Yay. Well, this is the fun part because next you have to do something that is pretty difficult and pretty annoying. Namely, you have to triage your crashes. Now, luckily, we don't care about exploitation too much because, you know, for us, exploitation doesn't matter because we care about whether we have or we don't have a vulnerability. We also don't really care about the severity too much because, again, like if the VM crashes, that's an issue for us, that's a vulnerability. We don't really submit to bug bounties. We just want the vulnerability fixed. And so for us, triaging is enough to say, okay, we have an out of bounds read or we have an out of bounds write. Yeah, we don't have to write a POC. And even though we have all these limitations that makes triaging much, much easier, it's still quite difficult because if you think about it, you know, you have your EVPF bytecode, that EVPF bytecode gets jit-compiled by Rust code. That Rust code then emits native x64 instructions. Those native instructions then somehow get mapped into actual process memory. And we have symbols for the first two parts and afterwards it's basically just good luck. If you take one of the crashes and you run it in, let's say, GDB, and you know, we just run it with one of the segmentation faults we found and we look at the backtrace, all you get is question marks. And that's not nice to reverse engineer because you basically have to exactly know what was written at that address, by which function and which instruction, literally BPF instruction cost us and so on. It's pretty terrible, to be honest. And the same with LLDB. LLDB, if we debug with it, we get basically the same, but it doesn't even give us a backtrace since we are just stuck somewhere in No Man's Land. But at least it tells us kind of what type of access caused the fault, like whether it's read or write and so on. And even worse, I didn't notice for quite a bit that actually on my machine, for whatever reason, LLDB had ASLR enabled by default. And so each time I was running it, it was crashing at a different address. And if you stared at like hacks addresses for 10 hours, you stopped seeing that they are changing and I just checked the first bytes and I was just like, what's going on? This code was different the last run and so on. Yeah, re-editing this is not fun, let's just say that. And so in the end, we still wanted to know what's going on. And so we started to customize the just-in-time compiler to basically on one hand give us the raw x86 binary, then give us a map of the eBPF instructions that cost it. And so we basically automatically annotated the eBPF instructions in GITRA to be able to see, okay, what may have cost us, spend hours and hours analyzing this. And found multiple out-of-bound reads and out-of-bound writes and so on. And again, we didn't care too much about exploitability, we just cared that they are there. So I can't tell you whether this would be somehow exploitable in the wild on the VM. But we wanted to at least try on a test chain to somehow get our validator to crash from the chain side. And this is again one of these points where you really should have read the entire code before you start fuzzing and get too excited about vulnerabilities because it turns out that the eBPF runtime contains something called a program verifier. And that program verifier happens to catch basically all our crashes. So we have vulnerabilities in the VM, but they are not exploitable on-chain. So this lowered kind of these severity for us quite a bit, but we still had some nice vulnerabilities. And so let's still, we need to go through disclosure. We submitted this talk, like we did this early in the year, started disclosure and so on. And on the 1st of May, we submitted to DEF CON. On the 11th of May, we basically got an email by the DEF CON refi board saying, hey, is this your blog post? And so it turns out that Edison from Secret Club basically fast exactly the same VM as we, using a different fuzzer and found obviously because it's pretty straightforward, basically the same things we found, some memory corruption vulnerabilities, and we basically had a full bug collision with Secret Club. The patch version fixed all our found vulnerabilities and we just told the refi board, hey, we didn't know about this, but this is not us, but it's very similar to what we do. And yeah, that was kind of a sad day. Congrats to Secret Club, 200K bounty is pretty nice. And our fuzzer is basically not useless, but yeah, it only finds floating point exceptions now. But yeah, I mean sad, but in the end, we still learned a lot. We still have a lot of takeaways. First off, just in time compilers are really a lot of fun to fuss. It's pretty cool because you have a lot of different pieces. They're pretty easy to fuss, which was kind of surprising to me because I always kind of considered it a bit dark magic because you hear about all these people fuzzing JavaScript, virtual machines and so on. Turns out with a small jit, it's actually pretty easy. Triashing it, however, is pretty difficult and can be pretty annoying. And also performance can really impact security. Sometimes you have to make a decision on whether you want performance or security. In this case, using a just-in-time compiler is somewhere in the middle but it's pretty difficult to get it right. But luckily, since then, they've added native fuzzing infrastructure to the whole VM and I think it's pretty secure by now. Also, if you find a vulnerability, don't get super excited. Verify that it's actually exploitable in the wild. And unfortunately, bug collisions happen and yeah, that's all I have for you today. I hope you liked this. Thanks a lot. Thank you. Cool. Hope you enjoyed DefCon and see you next year, hopefully.