 All right, I'd like to welcome everyone to my tech demo talk. And it is about curbing node size with bonsai trees and hyperledger basu. My name is Daniel Farron. I'm a protocol engineer and I am on the hyperledger basu project. I'm one of the original maintainers of the project. And one of the problems that we had with hyperledger basu going at Mainnet, one of the places we had a lot of our performance issues had to do with the size of the nodes. And anybody who's worked with Ethereum knows that the real test of scalability is can your node operate at Mainnet scales? On these are scales that are typically larger than most enterprise scales right now. So a lot of the work that we're doing on addressing Mainnet issues are gonna have great performance impacts on enterprise level issues. So to understand why node growth is such a problem, disk size, when I say node size, I'm talking about the amount of disk space that a node takes up as it progresses through the blockchain. You do need to understand a little bit of the details of how hyperledger basu stores its data. And this is dictated by Ethereum. So on each Ethereum block, there is a state group. And this is a Merkel hash of a Patricia tree, which is a bunch of fancy computer science words meaning we take the hash of a node, we put 16 of them together and we roll them up and we take that hash for the next level and we go down to each level. And we do these branches based on the address. So in this case, if we have an account 100, this would be the zero branch and this would be the one branch, then this would be the one branch and this would be the zero branch. So to get to a particular account in a tree, you would go down to this account 100 and all of the children locations below it are all put into this tree and all hatched up and stored to come up to a singular state group. So when Ethereum validates a block and sends it to everyone, it doesn't send an entire copy of all of the storage tree. It stores one mathematical number that is a hash that is a representation of all of that. So to keep that in memory, when you keep these nodes stored on disk, so we take the state root node, we take the first level of the root nodes and however many there's like five or six of these until you get down to the particular account nodes. And each account also can have state storage within its account. This is how a lot of the smart contracts and the RC20 tokens store their data is a store state underneath the account. So underneath each account, we similarly have a state root and then intermediate branch nodes followed by a final leaf node that contains the actual value of the node of the data that's being stored. So one particular state in time, if we look at just the left one, we have a collection of branches, a collection of leaves and this represents one state and the load impact for just one state's not too bad. It's at worst, double the amount of actual data being stored. But what gets interesting with the Merkle Tree node is you can go to another state, let's say we take this storage and we change it to 50 and account 100. All we need to do, all the new nodes we need to store are just the branches that changed. So here, if we change the state root node, we need to store a new value for this branch and a new value for the account and a new branch for the account and store that in the state root. And all these other branch nodes over here that we used, we get a key. And this is really cool because you can go back to all history in time and look at all possible states of all possible blocks. The problem is that storing these, I call this the forest mode. Storing this forest of trees is actually has a huge impact on the amount of states stored. So here we do, we just do one change on one data file. We have to update three different nodes to change one value. And it has a couple impacts and it takes time to read it. And it also takes up more space to write it. So one data value results in four different writes. Now, at the scale of one, you can see how it's big and kind of growing. But when you take it to mainnet scale, we're talking 150 million accounts, half a billion state roots, half a billion state nodes, it starts getting very, very large. And here's a chart from Etherscan, which is a very commonly used block explorer for Ethereum. And this is tracking over the past two years. So we have May, May, May. Yeah, this is just over two years, the growth of the entire block size of the forest. If you keep all of the forest tree nodes of an Ethereum block node, and here they're tracking geth and open Ethereum, but Hyperledger basically has basically the same growth curve because we're storing the same data. If you store all of the possible trees of all the possible history of Ethereum of all 12 million nodes, you would need today over seven terabytes of data just to store the data. And so there's a couple of things that impact that. Just the disk size storage is continually increasing at an ever-growing pace. You can see that the curve is getting higher and higher as growth increases, but also the speed of the access it. So to solve the solution, I took inspiration from bonsai trees. And if you look at both of these trees, on a DNA level, these two trees are exactly the same tree. It's a Chinese elm. But because of the way that it's manicured and maintained on the left, it fits into a nice small pot and looks a lot more well-controlled and a lot more useful, versus this Chinese elm tree, which is great for shade. It's great for outdoors. And if you have the space for it, just use it outside. But not everyone who runs Ethereum node wants to have these great huge bushy outside trees when they're maintaining a small, when they have a small place they want to maintain and manage it. It's a lot easier to maintain and manage a smaller tree. So it took inspiration from some of the aspects of the bonsai tree to basically help us figure out how to curb our node size. So the very first principle that I applied in moving to this new bonsai mode is to make it leaf-focused. When you're querying a value, you really only care about the leaf. The branches are just how you get there. So we split the leaves and we stored the leaves by location. And we put the branches in two different table spaces. And that alone provided for a good amount of database speed-up just because we have a smaller set to index. We go straight to it. But most importantly, we don't have to do any tree climbing. To get a value under the tree mode, un-traditionally, you would have to go down each level until you find the tree and get the pointer to the content because you don't know where the content's gonna be. But if we focus on the leaves, we store just the leaves by the location we can get there quickly. So we get rid of tree climbing by having it be leaf-focused. What do we do about the branches? So the branches I do, I keep them in a format I call it's well-manicured. We store the tree branches by the location on the tree, not by what the content is. And if you're gonna do it in forest mode, you can't store it by location. You have to store it by the content ID. And so when we store it by the location, we get an immediate speed-up and it has a huge impact on the database caching because the root tree node has the same node and the caching is easier to figure out that this root level, this block zero and block A through F and one A in the first 256, first two layers, they always come in with the same location. So it can do a much better job of caching this. But the second side effect is also created a little bit of implicit pruning. If you change a branch node at a particular location, you overwrite it instead of writing a new copy. So that's how we keep it well-manicured. We don't let these straight branches go out because they keep getting replaced by the new value. So that kind of has a little impact on some of the other usability of the trees when we store it only by the location is that we can't go back in time. And that is one of the key advantages of the Bonsai format is because it's a one point in time, we can very much handle this a lot easier with keeping the storage well situated. Now the downside of keeping only one point in time is we can't go back and prove historical state or access historical state without a deliberate design choice. And that's the next design choice that we need it to do. One, because people love to get block traces from months to years ago. And the second reason is because sometimes we need to roll back our blocks. Ethereum nodes are notoriously reorganizing all the time they don't have finality. So you need to go from one block that's gonna become an orphan and go back to another one. So to facilitate that, I keep what's called a tree log. So every time we transition from one a log from one block state to another block state, I record in that tree log all the values that are red and all the values that are written. And I keep that basically becomes kind of a two-way diff. And with that two-way diff, if you have a two-way diff to a block adjacent to the block that you're looking at, you can use that to roll back or roll forward to a different log state. So by keeping this tree state, I can go from one, I can go back three and then go forward two to get to the next block that I have. Another handy usability feature of this is I can go back a week, a month, a year, and I can take all these red and written values and I can just replay the block if they need it. The downside is we don't have the witness branches. So we can't quote a quote proof that this is the only way it could execute. So I call these tree logs more like a hearsay rather than a witness because we only have the end result. We don't have the proof to go with it. But if you roll back far enough, you can generate that proof to question how important that proof is. Or if you just need to know what the event's coming out of it and what the traces are of the logs. Now this log rolling is very important. Here's some of the demo, some of the graphs that I'm bringing up. This came from an early iteration of what I was working with it. We were taking a particular, we're syncing gorely, which is one of the smaller test nets. It's actually getting kind of big now. It's already up to four million, almost five million blocks. But because of the nature of gorely, it uses the click consensus mechanism, which you can see here that multiple blocks at a time at each level, if you go left or right, each vertical lane is a separate block number. And at each block number, multiple blocks get proposed by the various validators because the rules, everyone who is eligible to propose can propose. And on gorely, pretty much everyone who can propose does propose before they pick which is the most appropriate block. And there's rules to decide which one is the most appropriate block. But what that means is if you're listening to the consensus network, you get a lot of blocks that are gonna be orphaned that you need to roll forward to. So this actually accidentally provided an excellent torture test for the roll forward and rollback technology that I wrote into the bonsai trees. Because in the process of keeping in, for example, I don't know if you can see my mouse cursor, but there's a couple of blocks where we go a couple of white blocks deep before we go back to the gray blocks. And the gray blocks represent the truth and the white blocks represent the orphans. So each one of these white blocks, the bonsai tree node actually saw and actually processed and had to roll back. And if it's two blocks deep to get to the canonical chain and had to roll back at least two blocks and then continue rolling forward to get to the block that it needed to see. And this, you know, I set it up for days, weeks at a time. And it works great. It works really fast. And actually was quite surprised by how well it worked. So from all this information, some actual hard numbers to go with it. And these are numbers from a mainnet node. Once we got it working on Gorley, we started working on mainnet. And these are some of the more impressive numbers that I've seen from a project. I've been doing a lot of optimization on Basu and I was actually quite surprised and happy to see numbers of this quality coming from it when Korean actually got me the numbers for these. But this is a full mainnet block and we did something called a full sync, which is where we take the blockchain from the beginning of time and we play it forward all four and a half, five years. Every single transaction gets played and we keep all of the various immediate states. We don't keep any, and what's different between bonsai and forest is we don't keep the tree branches, we just keep the resulting state. And from the first line, the first set of numbers there, you can see the difference that makes. If we don't store the tree branches, we go from about 794 gigabytes needed to store the entire mainnet history that you could replay any mainnet block from zero up to 12,387,125. That took 794 gigabytes versus the forest mode where we need to track every intermediate branch and store that 12 terabytes is about what the estimate for bonsai was. We got tired of watching our AWS still go up and Korean just didn't finish it because we got the numbers that we needed. It was about on par within a multiple of what geth was going to do. The numbers we were seeing in the estimate is that it would come up to about 12 terabytes. So the storage there is on the order of 10x, 15x improvement between the two of them. So just the storage of running it on a long-term note. But I think the more relevant and the more impressive number comes from the fast-sync notes. And we had a node that I started up in February that was in forest and bonsai mode. And Korean checked to get in April to see what the difference between the two nodes were. The bonsai node that cleaned up its branches as it went from state to state clocked in about 470 gigabytes. That's with all the, you know, keeping up and doing the orphans and getting rid of the orphans and all the other things that go along with being a regular participating member of the network for two solid months. And the forest was already up to a terabyte of data. So there was, you know, at least a half a terabyte of data difference a little over that between a bonsai node and a forest node just running over a two-month period. And the part where that number is going up is actually consistent with the behavior we're seeing on Ethereum. Block gas limits are going up. There's more transactions coming in. DeFi contracts and other contracts on Ethereum are doing more and more interesting things with state, more and more complexities being seen in these contracts. So that the rate of growth is increasing on a per month basis is consistent with it. And so those are, you know, a couple of the numbers that are very encouraging for when bonsai trees becomes a shipped feature in Beisu, hopefully later this year. So those are the, you know, we gotta make sure it burns in, gotta make sure it's a good substitute. It's not shipping today. The goal is to get it to ship by the end of the year. And that's assuming that we don't get distracted by all the stuff happening in E2. That's actually a little bit of a higher priority. So yeah, so those are some of the hard numbers. Those are my demo. Yep. Cool. And it turns out that my time is up. So thank you all for coming to Hyperledger. Thank you for my demo. Thank you for coming and giving us a great presentation, Daniel. Thanks. Okay.