 Happy Day 3 of DevCon 6. Before we get started today, I just want to send a huge shout out to all the organizers, all the people that came from all over the world, and everyone who is at this talk today. So I appreciate it. So today, I'm Colin Myers. I'm co-founder of Oble Labs with Oshin. And today, we are going to discuss designing DVT for non-correlation. So before we get started, who is Oble Labs? Best way to think of us is that we're an R&D team. We're focused on building a different Ethereum infrastructure technologies for staking. Today, the core team consists of 16 members across eight countries, so we're quite distributed. Right now, our main focus is delivering distributed validator technology for Ethereum as a primary network to begin with. So before we dive a little bit deeper in today, let's kind of talk about just Ethereum as a whole and kind of the roadmap of scalability. So a lot of it, we just had the merge. And if anyone went to the opening ceremonies, there's this ossification discussion happening, which is essentially like hardening the things that we've put together, the things that we've created. And what we see here are the three main areas that we've been focusing on over the past few years to scale Ethereum into its next chapter. So on the left here, the first area is the execution layer. This is the most mature of all the different scalability technologies and research studies that have been done over the years. Cryptographic primitives that are used for this problem are optimistic and ZK rollups. This comes through in the form of layer two. It's actually been quite interesting this week to see that layer two is taking over as they should and layer two, honestly in my eyes, seems to be the future of what Ethereum will build out of from a research perspective. The second area ties to layer two and as it matures more, we are getting in front of the next problem, which is the data layer. So the data layer must also scale while execution scales. Common methodologies used for this is called data availability sampling and here are some examples of some few projects that are focusing on that. The third piece, which is kind of the most immature and not only like research and adoption and implementation is scaling the consensus layer. Like scaling consensus to date has basically been getting to Genesis and then getting to the merge. And now that we're in this post-merge world, we're all kind of reevaluating the consensus layer and saying how do we scale it more. Today what we're using is DVT as a cryptographic primitive and protocol to help scale the consensus layer. Now I'm going to take a moment to describe what distributed validator technology is in a bit more words. So at its most simple, a distributed validator is an Ethereum validator that runs across multiple machines. Why you might want to do this is it protects you from a lot of technical and social risks from validation. One of the first ones is software issues. If you're running just one node and just one software client, software generally doesn't run forever without issue. Similarly, running a distributed validator makes you more protected from hardware risks as everyone kind of expects, servers don't last forever and they die regularly enough. Going one bit further, if you take a distributed validator and you run it with other people, so each person only has a subset of the private key. That makes private key much more difficult, private key compromise much more difficult because you have to compromise three or more different operators to get the full private key to slash or do whatever with it. And the last one is a bit more abstract, but it's the idea of if your validator goes haywire, they become a bad actor, they've been hacked, remote code execution, whatever you'd like. Generally you have no safeguard if you just have all of your money with one validator, it's just you're in trouble. But if you have your money in a distributed validator, you have protection so long as only a subset of validators become Byzantine or go haywire or whatever you want to call it. Another thing that's super important is to talk about who benefits from DVT. The great thing about it is we think more or less everyone involved in validation can benefit from it. The liquid staking providers themselves, they at the moment are trusting huge amounts of stake and to single operators and they're just kind of crossing their fingers and praying that nothing bad happens. They would much prefer to be able to divide their stake across operator and then if anything bad happens to one, there's no risk or no loss to the liquid staking protocol. Similarly, the centralized providers can gain a lot from using distributed validator technology. I had the pleasure of building BlockDemon's ETH2 staking platform at Genesis and I know that you could reduce DevOps risk of you have to wake them up in the middle of the night when a machine dies if you have a distributed validator. Similarly, you can reduce your hardware cost because most people put a very small amount of validators per any given machine because if it goes offline there's nothing you can do but if you have a fall tolerant validator you can start to safely increase the amount of keys per node and that can reduce your operating costs and then on the more extreme or like the further out their side of things a lot of centralized operators either ensure their stake against slashing either on balance sheet or they go to a third party provider for insurance. This is currently extremely expensive and not even every provider offers it because the thing to ensure in a slashing is potentially you have to pay back everyone like all of that stake. So if you have an operator with hundreds of millions of dollars and ensure it has to potentially pay all of that stake you're gonna have a really high premium but if you were instead running a distributed validator and insurers says okay what's gonna happen if they get compromised? The worst case scenario is much lower and as a result the premium is hopefully gonna be much lower. The next thing I'm gonna do is wanna have Colin talk a bit about the market for distributed validators. Cool so how is stake accumulating? These were taken from Elias from Rated his liquid staking dashboard on Dune. Data in this area for those of you watching and who were looking at data this is probably some of the best data on the status of pools and how they've accumulated over time and I would highly recommend it. So where are we at today? This chart here starts on the left at Genesis which was December 2020 and it shows the dramatic increase in pooled assets as a percentage of total staked assets over time. So as we can see it's kind of quite frankly like exponentially growing. Today numbers where are we? There's about six million ETH locked in pools that equates to around $8 billion today and it represents nearly 37% of stake in the network. So the 37% of stake in the network is like probably the most important thing I wanna touch on as it ties to like what our thesis is which ties to how we believe DVT should be designed. We believe that over the next 12 to 18 months that pooling will become dominant inside of most networks especially Ethereum and we suspect that more than two thirds of stake in the near future will be inside of pools. DVT is today on the focal roadmap of pools to help decentralize and to help build themselves out. Therefore we believe that DVT saturation inside of these pools will also end up being 75% plus. Lastly, this is important because DVT is ultimately a security protocol. It comes in a middleware, however it's based upon security. So different than MEV Boost, right? You use MEV Boost to get more. With DVT you use it to protect the largest reward which is consensus rewards. We believe that DVT adoption will be kind of a winner-take-all market due to its security premises and the purpose of what it's used for. So to kind of set the tone before we go into correlation here, I just wanted to give a size of the market our expectations of how DVT will sit inside of pools and then how large pools will be as a percentage of total stake. Therefore if this thesis does play out, it's incredibly important that DVT stays helpful at scale instead of becoming more correlated and unhelpful. So now we'll spend a little bit of time on slashing, what it is and what it isn't. So, slashing. The best way to think of slashing for those of you as if it's a new concept is slashing is like the defense mechanism or the immune system of proof-of-stake networks. And what it's here to do, it's here to punish you for not following the rules in an attempt to keep the network credible and to follow the rules of the protocol. Today there's two primary ways to get slashed. The first one is a double sign. Back in proof-of-work this was called a double spend and this has been mostly predominantly what the top slashing event and the network to date has been through and we'll get into like how that's happened. The second one is a bit more abstract. Best way to think of it is walking back a slot height that you previously said was finalized. Now a more layman's term way of describing this is in the future you try to change something that you said was true in the past. So therefore you're trying to like change state inside of the network and if you do that you will be slashed for it. Inside of slashing there are different levels in degrees of slashing. There's correlated slashing and uncorrelated slashing. Uncorrelated slashing today is predominantly that the most of what's taken place in the network we've been fortunate enough on Mainnet to not see a large correlated event aside from maybe a couple small incidents and in an uncorrelated sense you can expect to lose up to a minimum of one ETH based on the state of what's happening in the network. When it comes to correlated slashing I won't spend a whole lot of time digging into it but what everyone needs to know is that in the event of like a network wide event correlated slashing can result in a validator losing all of their ether up to 32 which is much more of a catastrophic event. So as we begin to like look at the future of slashing there's certain technologies best practices that have kind of gotten over this uncorrelated slashing we'll definitely see a few in the network it's okay but now what we need to begin to prepare for is as like stake scales and there's more products and more infrastructure providers how do we prevent correlated slashing because it's like what could take the whole network down or take all a validator stake. Okay so what is an activity penalty? This is super important because it's not slashing. Many people who are coming into the space now try to say that inactivity is a slasher of a defense by definition it's not and the way that we should think about inactivity is if your machine goes offline it's okay you can take the time to bring it back up online and you can expect to lose what you would have made so there is a penalty associated with it but it's not a slashing and it's not because you broke the rules it's just due to downtime. Inactivity can also be more severe in scale so if a large portion of the network is inactive a big driving force in the E2 economics and design is this average percent uptime in the network so if that drops below 66 the inactivity penalty across the whole network will grow and grow and grow in an attempt to push ether out to rebalance it to get back to 66 percent so when it comes inactivity large inactivity events will result in sizable offenses for people still don't want to call them slashings but you can think of this mechanism as a way to rebalance the system to promote 66 percent of machines to turn their machines back on. Now we've discussed a bit what is distributed validator technology what it's useful for and then we had a bit of a refresher on slashing and inactivity now we'd like to discuss how we at Oble have made certain design decisions to minimize the correlation risk because as we said the title of this slide or this talk is how not to make things worse when we're trying to make them better so the first thing I wanted to talk about is the private keys because in a validator the private keys are more or less everything we've taken the decision to not allow any one entity to have the full private key that's not the customer that's not any one operator instead all the operators come together and they do a distributed key generation which means they all take part in a bit of a protocol and they contribute some entropy to it and at the end they all get their private keys and they get a proof that everyone took part and followed the rules fairly but there was never any moment where one person had the full private key for the validator this is super important because if somebody does they can run it in a validator and get you slashed we don't put these private keys more or less anywhere they're created on the machine where the validator will run we don't encrypt them and put them on chain or leave them anywhere where people could attack and compromise a large number of them to follow through with that once we've helped create these validator private keys we don't actually keep control over them Caron, which is the name of the software Oval has written is a non-custodial middleware which is a fancy way of saying it doesn't have the private keys it just asks the validator to sign something for it it sits in between the it sits in between the beacon client and the beacon API and the validator client and it intercepts the traffic going between them your Caron clients all they really do is they play a consensus game and they say, what are we going to ask the validators to sign and once they come to consensus on that they present that to each validator every validator says, okay, let me check my slashing rules have I already signed something at this height? No, okay, that's good have I said finality was further than what I'm being asked to say? No, okay, sweet I'm not going to get slashed I'll sign this message and I'll send it back to what I think is the beacon note we're actually sitting as a middleware there we intercept that signature we gossip them to the other different nodes in the cluster and once any of them have enough of the partial signatures they can reconstruct the aggregate signature and send it on to the network and the rest of the network says, yep that's just a normal validator, it looks good to me what's nice about this middleware and not having the private keys is if we are compromised, the worst we can do is show something slashable to a validator client but we get to rely on the fact that the validator clients are independent and have their own code probably don't have the same bugs and are not going to sign anything slashable most likely however, if you only hit just one validator client there is a scenario where you find a zero day in that and you can convince them to sign something slashable so one of the things we've been very focused on is being multi-client and what that means is you can run any of the validators that implement the spec specifically we're still working on support for all of them but you can use Lighthouse and Techoo already and generally speaking, in an enterprise operator like distributed validator you want to have a mix of all the different ELs all the different CLs all the different validator clients because if there is a bug in a client you don't want it to have a majority of your distributed validator because then it will sign and you might get slashed and the only data point we have for a client bug like this was when Prism had a timing bug back in the Medasha testnet and that was the most severe slashings across any testnet or mainnet today yeah, so the next thing I wanted to talk about is the networking side of things so over the last three design decisions we talked about we said, how are we going to set up the private keys how do we keep them maximally safe, step one then step two is how do we keep our software out of the driver's seat we don't want to have total control over these private keys because if we can sign arbitrary data that's a threat you don't want something running more than 33% of the network if they can sign whatever without any oversight the third piece is obviously our multi-client implementation trying not to have exposure to any one piece of implementation makes you more like resilient to bugs and makes you less correlated and then the last one is more on the inactivity side so the three before have been all about preventing slashing, preventing slashing, preventing slashing but we also try and prevent downtime of course if you're running a highly available validator the pitch is that you're going to be up more often than if you're on a solo note and similarly if we had a distributed validator architecture where something goes wrong and we knocked off every distributed validator all at once the inactivity penalties would be rather severe and we would make things worse when we were trying to make them better so how we have architected Karen's networking stack is they're totally independent from one another one group of people running a distributed validator over here has no common infrastructure with someone running it over in a separate place we think this is super important because it allows you to have non-correlation effectively if the alternative to this was if you kind of bound all of these together and to a shared like networking layer like message bus or gossip network the problem with that is well for a start it's usually slower you have to go multiple hops around the internet and with a distributed validator time is money so you don't want to be doing that but also if anything happens to this network you knock off all distributed validators simultaneously and that's something that we don't want to do so if you have a scenario where every distributed validator has their own independent network networking if anything does go wrong they go wrong separately and the nice piece about this is going through this like effort of making everyone set up new networking is definitely a bit more hassle but the benefit of having independent clusters is you can also have independent versioning and what I mean by that is one of the scariest things in software development is releasing an update because you're not really sure it's going to work and it's particularly scary if you're securing billions of dollars so the last thing we want to do is introduce a correlated outage by pushing a new version it is pragmatic to assume that you'll push a broken version at some point we saw guests do it last month and they're probably the most experienced team in the space so it can more or less happen to anyone you shouldn't assume you're just not going to do it so instead you're like okay if we did push a bad version like how can we minimize the issues and this is where having independent networking comes in if you have one cluster that's totally separate to another one you can update that one to version two and see what happens and if that's all healthy great you can update a second and a third and a fourth and you can gradually roll out this update across the network and if anything goes wrong you can kind of abort and you can kind of panic if you go the other way of we have shared messaging and you know everyone has to talk the same protocol you have to do what the kind of ELs and CLs do which is you pick a slot number and you release a new version and you're like add everyone on Discord you pray to God that they have running the new version in time because if they're not running the new version they go offline and you also pray that the version is right because if there's nothing wrong with it you knock everybody else off as well so yeah the inactivity side of things the penalties aren't as severe but it is also super important to us when designing distributed validators not to make things worse when we're trying to make them better and I'm going to hand it back to Colin for the last few closing thoughts cool yeah so to close it down again appreciate everyone coming out DVT is this I guess well we've been working on it quite a long time but since ETH Denver at the beginning of this year it's really now we've begun to have its turn essentially of adoption our North Star is a project today what we're most focused on is convincing everyone that validators are communities not individuals today the way that the network topography sits is validators are a single key validators are a single person or validators are a single entity and with this technology we can migrate that paradigm into validators being run predominantly by communities so we have about five minutes left again since DVT's kind of been this new topic that's been introduced over the past few months we want to make sure we take time to answer anyone's questions and again thanks everyone for coming out pushing and incentivizing people to have diversity in where the machines are is actually the real problem there right there's a variety of validators large ones that they choose to run on-prem there's a variety that they choose to run in the cloud we were at dinner last night and the menu was running on US, Amazon, East too and we just laughed because like everyone's running on US, Amazon, East too so there are like other types of correlated problems with internet usage that we necessarily can't fight but still susceptible to them I was wondering about the DKG itself about the ceremony and how do you think about the risk that there is some problem some data centers or something and somehow like at one point two out of four keys get lost like how probable do you think this is and what do you think about possible mitigations to this problem yeah this is a very good question and this is usually referred to as like verifiable secret sharing in like academia and this is the thing of doing a DKG is one thing but how do you know it actually worked how do you know that you know everyone actually contributed everyone got a piece like they expected and usually there are two types of ways to prove a DKG is fair they're called verifiable secret sharing and publicly verifiable secret sharing but ultimately speaking at the end of a DKG everyone that's taking part they get a private key and they get a proof in a verifiable secret share you need to have one of the private keys to check the proof in a publicly verifiable secret share it's like a zero knowledge proof anyone can just check that it's verified so yes it is actually super important to make sure that everyone took part and usually there's like challenge rounds built into DKG which says you know does everything look right does anyone hit the abort or is everyone going to like sign off on this and yeah at the end of a DKG normally there's a proof I think right now we have a VSS proof meaning only the peers know what was legit but ultimately we would like to add a zero knowledge proof so even like you know the is yet Lido can say yeah those four operators definitely did the DKG correctly because they wouldn't have been able to produce a DK proof otherwise for sure so the question because he's not miked up was you know what happens if people lose you know more than a subset of keys and honestly the answer is if you lose you know more than the threshold you're probably gonna you're probably out of luck to be perfectly honest so generally speaking that's why you want four different operators and they should all make you know independent backups if you have two operators that didn't make backups and left them in the same data center and everything blew up yeah that's probably a big problem to be perfectly honest and yeah that would be one of the ways where forced exits would be nice because then the person that the withdrawal of key could be like oh please send it back to me but what would realistically happen is right now you'd be offline and you'd be offline until you hit 16 ether and get kicked out so yes if people don't back up their private keys you know even in a distributed validator if you lose you know too many of the private keys you're still screwed