 Hi everyone, this is Brady from Sentinel. Today we'll talk about how we made Liddy around 15 times faster. So just on background, Liddy is an essential software for indexing the Falcon blockchain. It provides data extraction and analysis compatibility. A few things that Liddy can do is to, for example, extracting blogs, messaging receipts, and the actor state changes such as minor sector events, market deals, and FEVN data. So the main problem with Liddy previously was the infra cost. And that's probably the top complaint for our users. And because some certain tasks that requires a lot of resources and time to process the data, and once the time exceeds 30 seconds, because that's a duration of an epoch, then we will not be able to keep up with the change. So in this case, we will need to process the task in parallel. We need to run multiple Liddy nodes in order to keep up with the change. And this can increase infra cost significantly. So for example, a five node cluster of Liddy nodes will cost more than $10,000 a month. And in the past, one vendor even quoted us like 1.5 million for two years of contract to operate Liddy and the database. So here's the diagram of the Liddy architecture. So Liddy is a specialized Lotus node. We import the library from Lotus, IPOD, and Actors, et cetera. So the main components are the indexer, processors, and the exposure. And here's the diagram of the distributed worker pattern. Like what I mentioned, when we need to run multiple nodes, we need to set up this infrastructure. So it contains Notifier, the message queue, and workers. So although this pattern will scale horizontally, but it will get expensive really quickly. And I was surprised when I saw this. And it requires a distributed system in order to parse and to extract Falcon data. But I was told that it's because the state comparison between epochs is very expensive operation. Although I'm not convinced, so I decided to look into it further. So we start investigating into the performance bottleneck. So we try to start with the most time-consuming task, which involves market deal data. So you can see that here, the task process duration for market deal proposal is more than one and a half minutes. So we enable the tracing feature in Liddy. So here's a simple trace for processing one tip set in Liddy. So the total time spent for processing one tip set is 1 minute and 33 seconds. We spend most of our time on deal proposal processor. And within the deal proposal processor, you can see we spend all the time doing amt.dev. So what does that mean? So amt is an IPOD data structure to store an array of data. And amt.dev is to compare the difference between two arrays. And then below that, we can see we also spend quite a lot of time on export data, which is the actor states. We spend 20 seconds to just persist the actor state data. And further drill down, we realized that we are making one database code per row, which is definitely not right. But it's an easy fix. So in order to understand why it's so expensive to do the amt.dev, I reviewed the coding of the amt library. And eventually, it discovered a bug in the library that's causing unnecessary tree traversal. So in order to explain that, I draw a diagram to demonstrate how we compare to amt arrays. So an amt is actually a tree structure. So each node will have links or the actual values. So links will point to other amt nodes. And there's a CID associated with each link. And the values are only stored on a leaf node. So here, we can think of the V1, V2 as the deal proposal. So we have a array of deal proposal. So we are comparing the proposal between two epochs. We have state one from previous epoch and state two for the current epoch. So we are comparing the amt. We look at a CID, associated with the link first. So CID is calculated as a content ID. So it's calculated based on the content of the subtree So if the CIDs are the same, it means that the subtree has the exact same values. So we don't have to compare further. So in this case, when we look at the first link in the root node, we saw that both CID 1, so we stopped there. And then we look at the other side, we saw CID 2 and CID 5. So we know that the content of a subtree is different. So you have to find out the actual different value. We do it recursively. And eventually we'll find out that in one of the nodes, the value of V6 is changed to V7. OK, so this is how the MDF should work. But the bug I discovered is that even though when the CID is the same, we still try to navigate down into all its children nodes. This is especially bad because that means we are loading every node from the disk into memory and just to find out that the values are the same in the end. So that explains the high CPU and IO usage whenever it involves a huge state comparison. So a similar bug is also found in the IPOD HEM library. So the fixes for both libraries are simple. And we also optimize the active state persistent to batch insertion mechanism. So the result, market deal proposal extreme time reduced from 100 seconds to 25 seconds. Active state persistent time reduced from 20 seconds to 0.2 seconds. The time to process when you pop reduced from 100 seconds to 6 seconds. This is especially meaningful because once the process time for when you pop is lower, it's shorter than 30 seconds. We no longer need a cluster setting. So we can now run everything on one single instance. And the mounting for cost is reduced from $10,000 to $100,000. And I think now everyone is very affordable for everyone who wants to run, who knows on their own. So next I will talk about impact to staff. Yeah, thanks, Bertie. So all of these massive improvements have resulted in very positive feedback from key Lili users such as Ngram. They're now able to use Lili and Archival Snapshots to backfill FEVM data, specifically contracts data. Not only that, since Lili is performant now, we can add more complex processing tasks instead of doing this in a separate data processing pipeline, which I did previously to Bootstrap. So thank you to Terry for baking that into Lili. So now our Lili node operators are able to do that themselves. Without having to create another service or just extracting contract data. Additionally, Starboard, another one of our key customers has also has been very satisfied with their performance. And they expect that the cost of running Lili nodes will be more affordable for them going forward. And that hopefully reduces the dependency between them and us. Moreover, because of the reduced batch processing time, this also means actual financial cost savings, not only for ourselves, but also for node operators and actually also for our batch processing job that we have in-flight at the moment. Here are some resources from our team. Bertie has written a write-up for the performance optimization for people who are interested. Our Notion page, our roadmap, and also we have started a new RFC documentation to better collaborate with other teams and other PLN companies. And you can reach us on Phil Santanel's Slack channel as usual. Thank you.