 Good morning, guys. Thanks for coming bright and early. So I'm going to actually tempt the demo gods today. It'll be interesting, because the things that I'm going to show you today and demonstrate take long enough to deploy that I kind of want to get started right now. So I also want to thank the conference organizers for providing us with these free, LTE connections, because that's what I'm using right now to be able to do this. The Wi-Fi hasn't been great, but fortunately we have these wireless hotspots we can use. I'll just deploy this and then break into it in just a minute. So I'm changing my config params to suit my needs, then going ahead and setting up my dependencies. And then I go ahead and kick off this cloud formation template that will deploy the solution that I'll be talking about right now. So let me jump back into that. I also want to say hi to one of my local counterparts from Tokyo, Yuki Nakatake, which is right here joining me. Thanks for being here. So I'm Carl Youngblood. I am a senior blockchain solutions architect with AWS. And I've been doing work with blockchains for quite a while. I first became interested in blockchain in 2010 when I read Satoshi's white paper. And it was one of the only white papers that actually came with working software. It was really exciting to actually set it up and run it. And I was lucky enough to be in living near London in 2014 around the time that the first Ethereum meetups got started as well. So before I joined AWS, my brother and I co-founded a consulting company called BlockScale. And we deployed some node infrastructure solutions for companies like MyCrypto and the Tezos Foundation. And I'm going to be sharing some of the things we learned in that process today. So first, I want to share some thoughts about decentralization and how it can sometimes differ in theory and practice. So and after that, I'm going to explain a strategy that we used for deploying highly available node infrastructure. And then some ways that we mitigated against some different challenges and problems with EVMs. And also how we made our deployment more cost-effective. And then finally, I'll show you a little bit more of this solution that I just deployed. So one of the great promises of blockchain technology was that it enabled greater decentralization. So in the same way that anybody, the internet is a permissionless network that anybody with a computer and the right equipment can just hook up to. And as long as they're speaking the right protocols, they can join in this network. Blockchain also eliminates the need to trust in these large central authorities to track information. Even private permission blockchains have the goal of leveraging decentralization across all the participants to increase the security and the reliability of the network. But some of the initial ideas about decentralization have been kind of naive. So Vitalik, his earlier claim that Ethereum should be able to achieve on-chain scaling while running on nothing other than consumer laptops has been forced to confront some perennial challenges in computer science about data locality and constraints of networking. So while that scaling work is ongoing and I certainly applaud it and am grateful for their work there, companies still need practical blockchain solutions for deploying their infrastructure that are robust and reliable enough for their applications right now. So some companies rely on third-party RPC services and these are very helpful and they definitely have a place but they also represent a significant reduction in the overall level of decentralization in the ecosystem. So there are times when you're building an application that's important enough that you want to reduce your reliance on external dependencies as much as possible by deploying your own infrastructure, something that's fully under your control. And then you also have the added benefit when you do this that you're actually contributing to the overall decentralization of the blockchain. So in practice, we've kind of observed that the best way to birth tomorrow's decentralized infrastructure is to build on the reliable solutions of today. So if you need your DAP to be highly available, the underlying node infrastructure that it relies on needs to also be highly available. By leveraging the services of an experienced cloud provider for your blockchain deployments, you're gonna be able to take advantage of just a lot of different things. So they're unprecedented scale and efficiency and global reach and also the cost savings that come from like the pay as you go model instead of the sort of over provisioning your resources for periods of peak load. So before if you're doing an on-prem solution, you have to kind of over stock up on resources just in case you need them. And clearly this is less efficient, it results in a lot of waste. And one other huge benefit is that these cloud providers like AWS have been doing this for a long time, they really know what they're doing and they're already running on over 50% renewable energy and our AWS is committed to getting that to 100%. So that's kind of a nice bonus. So I'm gonna talk a little bit about how the infrastructure works. The AWS cloud is divided into several regions and each of them has something we call availability zones or AZs. So they have several of these. And these AZs consist of one or more discrete data centers and they each have redundant power and networking and they're housed in separate facilities that are on stable flood plains. Though they're geographically isolated from each other for disaster recovery purposes, they're also close enough that the ping times between them are very low. So the RPC service deployment that I'm gonna be showing you today distributes Ethereum nodes across different AZs to ensure that the service continues to function even if one of the AZs suffers an outage. So I'm gonna go through some of the architectural approach to this deployment. First we have requests that are coming into the Amazon API gateway. And this gateway checks to make sure that the requests are recognized and it blocks certain forbidden methods that are not really needed for everyday use that might incur additional overhead. So it kind of keeps your waist down. And then if the call is valid, we pass it on to a network load balancer and that serves the request to several container instances. And each of those container instances is running in a separate availability zone. So in our experience, the primary bottleneck that you incur when you run nodes is around IO. So these are storage optimized I3 instances. They're optimized for very low latency, high random IO performance and they each have like an SSD based storage that you have access to on the container. So to optimize the resources, we run several processes or tasks per container depending on the container size. And that technique is sometimes called bin packing but it's the idea that the container size you have available to you may not be the perfect use of the resources that each node requires. So we add, we choose how many nodes we want to run per container. And these ratios that I've listed up here are in our latest testing. These are the ones we recommend at the moment but they could change as blockchain sizes increase or change or if the efficiency of blockchain storage changes. So one particular challenge that we ran into is how to get around the incredibly long sync times for new nodes. So if one of our nodes becomes unhealthy, we wanna be able to start up another node quickly. And with Geth taking several hours like almost a day to sync a new blockchain and parody taking several days, we needed a better strategy than just starting our nodes from scratch and having them sync with the blockchain. So we created a special task on one container that we call the updater task. It's down in the corner there. And this task runs a node that's periodically taken offline every four hours or so and copied to S3. And when you copy within the same region to S3, it doesn't cost any, you don't incur any bandwidth charges. So when new nodes start up, they copy the contents of the blockchain state from that same S3 bucket to their local disk before they begin. And then this enables them to start up within a few minutes. If we're lucky, it'll be like actually running by the end of my talk here. So let's take a look then at the code for just a sec. So I've got the repo right here. For those of you who, I'll leave it there for just a second longer. For those of you who want the link. Of course, this will be published later. And in here, I'm a strong believer in a project having all the information that it needs in its readme to be able to reproduce it and run it and execute it. I think Chris Wanstrath, one of the GitHub founders called this readme-driven development. But anyway, I think this is important. So hopefully everything you need to be able to run this is there. And I'm gonna just bounce over to my AWS Management Console to just see what the state of this thing is going. So we launched this additional stack. It looks like the connection is so slow that it hasn't even fully kicked off here yet. So it's probably not going to be finished by the time that I'm done. So for that purpose, I prepared a video just in case this was going to happen. I kind of had this feeling last night that I might run into this. I tempted the demo gods. So I'm gonna bounce over to kind of what you would see as this progresses a little further along. So that kicks off this cloud formation template and you get this result saying, okay, we've just started this. And I've got a service that I've already pre-deployed that we can actually test against. So we're seeing some of the RPC calls now going out to this node cluster and they're being responded successfully. And because I have another node cluster running already, I can actually do that as well. So if I want to, for example, I first need to go to my API gateway and find out what the URL of my service is. So we'll do that. So while this is loading very slowly, I can take you through some of the things that we're still working on in this. So this right now supports the Parity client. That was one of the, you know, at the time we first developed this, Parity was a little more stable, so that was what we were focusing on. And I am very close to releasing to merging in a PR that will support geth as well. But it's actually pretty easy because of our approach of just kind of copying the contents of the blockchain state off disk and syncing it with S3. It's very easy for us to kind of support whatever node service we want to. So as, you know, ETH 2.0 comes online and a lot of new EVM implementations are coming out. This shouldn't be too hard. So let's see where we are here. So grab that URL. And we get a successful result very slowly because of the network. It's not the service, I promise. Anyway, so this has been a very stable, successful deployment that's been running for some of those companies that I mentioned earlier for quite some time. We continue to maintain the Tesos Foundation's node infrastructure in a similar way. And happy to answer questions about how we did anything and talk with y'all offline after this. But thanks a lot for being with me today.