 We're going to go in and get started. Hi everybody, I'm EG, and this is G. And we are members of Consensus Working on the Hero Project. Woo! Woo! Today we're here to talk to you about our open architecture initiative. It's really great to be back here at the, oh, yeah. All right, well, it's really great to be back here at DevCon. This place is always going to hold a special place in our hearts because we launched at DevCon in Shanghai just over three years ago. And obviously, the Ethereum ecosystem has seen a huge amount of growth in that amount of time. Our team has grown. Our architecture is pretty much unrecognizable now versus what we had three years ago. And that's really what this talk is about. We entered into this space because we saw the troubles that people were having with trying to enter Web 3 and explore what could be done with building smart contracts and building things on top of Ethereum. And we thought we could help with that. And one of the things that we want to be now is share our lessons learned over the last three years. Talk about what worked for us, what didn't, and just continue to share our experiences and lessons learned going forward. And that's really what this open architecture initiative is about. So talking about single-node architecture is one of the, whenever we all got involved with Ethereum, involved with blockchain, you all start at the same place. You start running in the Ethereum client. It connects to the network. And you start throwing some requests for data at it, whether that's giving this list of transactions, let's get back to this smart contract. You're interacting with a single-node that's typically installed on your laptop. Maybe you start working in a co-working space with some people. And you start just sharing this and then because nobody wants to have all of that space taken on their Mac where you can't upload or upgrade your hard drive. So what you end up having is the single-node coming on the single-chain. And maybe the amount of requests that you're doing out there are starting to be slower than what you would expect them to be. What latency starts to increase. And you end up trying to do traditional wet-scaling tactics, which starts with rhythm-leasing. So that just means throwing more server resources at it. But that only takes you so far. At some point, these server resources scale beyond what you might have access to, but it's in which. And what we've seen is that developers end up building custom solutions around this. Whether that's custom catching, custom APIs, a bunch of sort of like scaffolding architecture around this. And what we've seen is that there is now like this inequality of data between different development teams, depending on whether or not a specific team has hard expertise with like wet-scale or if they have beyond the resources on time to put into this architecture. And so that's what we're trying to do is share more information about what's worked for us. There are different types of user profiles that use Ethereum nodes. For the majority of users, all you're doing is interacting with a smart phone track, checking that token balance and selling a transaction. You're probably using a lifetime, or you can be using a lifetime. And that's currently sufficient. If you're a developer that have fun and you try to do something that is relatively simple, you think you're probably still using a lifetime. It's only when you start trying to do slightly more complex things with that data, like looking at historical information on chain, that you start needing to run like a full-net or a whole archive node. And maybe at some point you turn that into a business that is more focused on watching analytics. And at that point, you need this node to be always on the line and available because maybe you built an API on top of it that your users are consuming. And all three of these types of user profiles are using the same underlying Ethereum client software. Ethereum clients were meant to be ubiquitous. Everybody was supposed to have access to the same software and be given the same opportunity to access this data. So whether you were in a gap or parity or can't be on an LHIP ledger basis, all three of these user profiles here are running the same software. And it's not necessarily tailored towards their use base. So when you think about an Ethereum client as a model like a piece of software, think about this as like a Web 2 application. Like maybe this is like a Django app, or a medium-range app or something like that. The way that people typically solve web scale was by looking at this thing as subsystems. So what are the subsystems of an Ethereum client? There's the query interface, which is your JSONRPC web server. There's the peer-to-peer connections from network. There's the EVM that's actually able to process new blogs and transactions. And then chain data storage that actually stores that information locally. And then the transaction system that actually receives these transactions and propagates them out across the network. Talking about something like the chain data. So if you're running this on a single piece of hardware, how do you scale access to chain data? How do you pass what a specific method has? For depth, you're going to be using level DB for parity to use graphs DB. By taking this and thinking about it, not as this model like a piece of software, but thinking about it as subsystems, we started to apply just traditional like web scaling to this because we're not running this on just like a single Mac, but we're running this in the cloud. And so you start with or at least it. You put this on a server, it's maybe using eight cores, 60 gigs of memory. And these are usually pretty proficient. Then you can get to over 8,000 requests per second because we're running one server if you start to let them configure it correctly. And maybe you can get double what you started with for certain things that can just double the CPU or double the memory. But then beyond that, that's where people start running into issues. And so we have what we call our cloud type architecture. So we conceptualize this as taking those subsystems and being able to optimize and do the current thing. So you don't just say, this is level DB on a single instance. You say, this is the chain data. And if I want to be able to create this data more efficiently and retrieve this data more efficiently, I can build additional indexes of data. Maybe I take this and instead of putting it in level DB, I put it into Postgres, so I put it into DynamoDB or something like that that has like web scale behind it. And then you can index the data specifically tailored to your needs. And you can then do this with any other subsystem, turn it into a specialized microservice if you're towards your application. All of these things start with having that reliable node connection. And that's how you can quickly provision it. If you're trafficking that data on a client, it's much simpler than it used to be. It used to be on 2016, you didn't need extra tooling. You could work sync a Paraly node in minutes and you could get a get node in like an hour. But over time, that's grown in terms of donation as chain data has grown. And so the first thing that we tried was doing simple EBS snapshot. So if you're familiar with Amazon web services, there's this concept of elastic block storage, which is the ability to snapshot periodic backups and quickly spin up new disk types that can be associated with any server that you spin up. So instead of taking, say, 12 hours to sync a node, this thing can be up in online like an hour. And so additionally, these periodic snapshots gave us failure recovery. There were cave-in certain upgrades that corrupted chain data. And we had to find a way to quickly recover otherwise our servers would have been down. So that was one of the things that we started with. Lately, there's been a lot of improvements to sync times where it's gone from 1 and 1 half hours with get1.8 down to four hours with get1.9. These are, this is test data from the get team. They did great work in the center. And we're able to get these kinds of improvements specifically by optimizing IO utilization. IO utilization is really important because when we run these things, they use a huge amount of IOPS because watching data in terms of being stored in key value stores in the way that the data is verified is very read and write heavy. So we actually use this type right here, the i3-4x large, which gives you 825,000 random read IOPS available. If you weren't using this and you were using PES storage, you were going to be limited to a fraction of that, like one 10 of that is what you really have. This only gets you to that four hour mark and that's not fast enough if you need to quickly spin up and you know for a minute and breathe. And so we ended up combining this approach, combining periodic backups with being able to sync that directly to any node when we spin it up. Thanks to my crypto team for sharing their implementation of this. They were the first ones to really try this out in production and they share their implementation here if anyone wants to check it out. But this now got us to a point where this source of truth node is what we call it. Syncs from scratch, is able to sync from scratch and how long it takes, either a fast sync or an archived node to sync. Then we have a landing function that fires off every hour to backup that data to an S3 bucket. We can generalize this to any cloud that offers a similar tool. And then lastly, if we need to spin up any capacity to upgrade the node, these nodes have a high three-put pipeline between S3 and EC2 and it's fully able to saturate that IOS that's available to the server and get this node up and running about 50 minutes from the time that we started to the time that it's ready. So once you have this reliable node, you can start building additional services on top of it to optimize your traffic. And it really needs to be tailored to the traffic patterns that you and your application and your business have. So G's gonna walk you through the traffic patterns that we see. Can everybody hear me? Yes. I've memorized our traffic into three different types, near head, archived and full. Near head is any request that's within about 100 blocks behind head. Archived is requests that require approved data and then full is requests that do not require approved data. As you can see, the majority of traffic that we get is near head. It's fairly consistent whereas archived data can fluctuate anywhere between 26 to even less than 1% depending on what our users are doing at that given point in time. So the first thing we did was we passed our near head data using CDN. It worked really well but it just came with certain limitations. For example, we could only cash get block by number and get block number. It also wasn't really scalable. So we made a decision to store our near head data in a red CD and it was much more scalable and also allowed us to store all of our near head data. In addition to our near head cash, we have a set of specialized micro services that either index or cash like specific types of data. The one I'm gonna talk about today is our log indexer which services are get blocks request. The way that Ethereum clients retrieve block data from the get barely time consuming, it retrieves every block, compares it with a blue filter, then it retrieves the transaction for the match blocks and then again, compares the generated logs with the filter. It works really great for a person that's just querying their own blocks using their own node but we get up to like a hundred million get block requests today. So when that happens, it overburns our Ethereum nodes. It also increases the response times for our users. So we actually index all our logs with incoming new blocks. One of the issues that you can run into when you do this is to correctly account for rewards or you might be servicing incorrect data. So the log indexer actually uses another micro service that we have. Sorry, called re-work tracker and re-work tracker gets dead in blocks. So if we have a block five million, it outputs that data to a consumer in which case is the log indexer. The log indexer then can push the events to its DB but we use MongoDB and continues on with block five million and one. Log indexer updates that information and then when we get block five million and two, re-work tracker then realizes that the parent hash of the current block does not match the block hash of the previous block that it receives. So it actually, and it's a rollback event which the log indexer then consumes to delete that information. And then when re-work tracker retrieves the correct block information, it can then make sure that it has correct logs for block five million and one and then it just continues on with the incoming blocks. It's fairly simple but, and we use it right now to index logs but it can actually be used to index any kind of data that's associated with block information. So this is a fairly simple view of how we handle our requests. The majority of the traffic that we get gets handled by our near-head cash layer and then we have our set of indexers and caches that handle the more frequently called requests and then whatever is left is then service choir theory gets. So those are some of the microservices that we use to help handle the large amount of traffic that we get but we also have a set of internal tools that we use just to help maintain our infrastructure. One of the recent microservices that we have consourced is key router. It maps keys to a consistent set of nodes. It helps route specific kinds of traffic to a consistent data source. A member of our team, Andrew Petrov, created the VIP node which is like a way to incentivize the use of live clients but we actually use it to manage peering or nodes. If anybody's interested, go check it out, it's pretty cool. And also we have our node monitor that monitors the health of our nodes. It tracks like latest block number and then compares it with an external source like ether scan to make sure that our nodes are near-head. It also monitors like clear cow and whether or not it's sinking and that all helps to make sure that the peering nodes are healthy for our users. And then I'm gonna head back to Michi. Yeah so this is really just the start of our open architecture initiative. Having this source of truth infrastructure, having some standard way of indexing is really just the start. Over the next few months, we're going to continue sharing more information about how we've optimized specific parts of your client's subsystems. And one of the things that we wanna do is continue working with partners in the ecosystem to continue to improve this and share knowledge so that we can keep building methods going together. And we always wanna do it in a way that's open collaborative and standardized. We never wanted to hit for the API that was non-standard and server bots, people want to be just one provider, one specific service. And so the next thing coming for us is the tracing API. If you've ever tried running the tracing API available on your local node, that's something that we're going to be building on top of this, well we're already building on top of this and we'll be available in the next couple of months. Outside of this, this architecture is generalized. It's not specific to Ethereum. It's definitely easy to support EVM-based chains using this. We'll be adding multiple chains in the near future. So if you have any questions, we're gonna be doing a Q&A session from 9 a.m. to noon tomorrow. And at the consensus booth, please stop by, ask us about, if you're a plus, this open architecture initiative and just really come by and chat. You can have these great debt-confined limited-edition cookies available if you want to come and check it out. And a couple of members of our team are gonna be in the back, handing out some swag after this. We want to say hello to Tim and Andrew. And that's it. Thank you.