 Hello again, so today we have the first panel of the day here in the tropical sunset the scenario So I'm very happy to introduce you to Adrienne Suton, Oisin Kain, Paul Hauner, and Vasily Shepavolo Yeah, they are going to talk about its 10 p.m. Do you know where your name Nordic is? Sweet Thank you very much everyone as some of you that can count might notice we're one short in a panelist But I think he'll join us momentarily, so I think we're more or less kind of kick off and do a bit of intros and stuff beforehand And up there is the man himself Vasily Okay, and Probably with slides sweet, so today we have a panel on Valid air mnemonics and their and their security It's been almost two years since the beacon chain launched and there's quite a lot of people that haven't touched those private keys in quite a long time Over the course of the next 50 minutes We're gonna look back at you know what's been done on the key security side for the last while We're gonna see what things are looking like right now in terms of validator key security And then we're gonna look a bit forward towards withdrawals, which I'm sure lots of people want to know about But before we get to that I would like to do a few introductions Okay, and oh, yeah now sweet so briefly. I'll introduce myself. My name is Oshin kind I'm co-founder at oval labs. We build distributed validator technology Which is the ability to run one validator across multiple machines by multiple entities And yeah, that's my little quick intro. I'd like to introduce Paul next. Hello, my name is Paul co-founder of Sigma Prime I've been working on Lighthouse, which is an interim consensus implementation since 2018 Yeah, I've done a lot of the initial development of it and now I do a lot of kind of overseeing and reviewing and still some development I'll pass it on to Adrian. Hi, I'm Adrian Sutton I'm with the consensus and I started out on the basu team doing the execution layer side before there was an idea of a consensus layer side Spent about a year and a half pretty much full-time on basu and then switched over to Tegu because I heard that beacon chain was the new hotness and Tegu is a consensus client So I've spent the last. Oh, I don't know three to four years Helping build that and bring the merge to fruition. It's been a lot of fun I've been working with people like Paul and a lot of fun there Yeah, and Vasity Hey, I'm Vasily Tech lead at Lido also, which is probably more relevant to To this panel. I used to be CTO in a p2p validator So I know I think the tool about managing keys for validation. So Sweet Awesome. So we're gonna like test that and figure out some of the good bad nuggly about managing private keys And so, yeah, so the first thing I want to do is look back a little about how it started and figure out, you know Some of the basics about what is a mnemonic and stuff. So I thought Paul, would you explain to us what a mnemonic is? Oh, yeah, sure. So I'm not sure I can get a textbook definition, but a mnemonic is series of words of phrase Say 20 12 to 24 variable length. I think it's I think 24 is a standard in in f2 So a bunch of words easy to remember supposedly And then from those words you can you can generate Any number of a validator keys. So there'd be signing keys or withdrawal keys. So the premise is that Once you generate this string of words Then you write those string of words down on a piece of paper and then Use those words to generate your validator keys and you put those validator keys on On some hardware on on your hot machines And then you only keep a copy of the mnemonic On paper So that yeah, so that in the future if those keys were to be like if those boxes exploded disappeared You could use that mnemonic to generate exactly the same keys again Sweet thank you very much and I wanted to dig in a little bit You mentioned withdrawal keys and hot keys. I was wondering Adrian Could you tell me the difference between them and do they have to have the same seed phrase? Yeah, so The the hot keys that we talk about are your validator keys and they're the ones that you will have used and seen so far In dealing with ethereum and if you're running a validator, so they're the ones that are on your node And your your validator client has to actually load them into memory Any kind of secret key that's loaded into memory and hot on a machine is Not as secure as something that's locked up in a safe So they are at a higher risk of being stolen. You still do your best to Secure them and secure the machine obviously But we wanted to make sure that there's a a big difference between those hot keys that That run your validator and sign things and the keys that actually control the money and and whether you can withdraw it so From your mnemonic by default you would generate a validator key using one path and as paul said you can generate multiple keys So from a different path you get a separate validator key And it's that validator key that you will use once withdrawals are implemented to set up withdrawals and be able to To actually recover your money again Do they have to come from the same mnemonic? No, they don't you don't even have to use a mnemonic You can just generate the keys directly without going through the mnemonic process But typically, uh, it's a lot easier to back up one mnemonic and not lose it And etch it into steel or whatever it is and store it in a safe well then to have multiple keys that are You know being managed and and able to be lost because if you lose your withdrawal key That's it. It's game over. You're not getting your 32 week back Yeah, that's kind of one of the things we're going to get to a bit further on So as you mentioned in the seed phrase thing, um, a lot of people that are a home staker would have used the eth2 staking cli That set up, you know, one 24 word mnemonic that had both your withdrawal keys and your hot keys. Um It's unfortunate. We were supposed to have one more panelist jim mcdonald And he's done a lot of work in the specific problem But uh, the reason that this thing is titled it's 10 p.m Do you know where your mnemonic is? Is because some unfortunate people have had these mnemonics compromised over the last two years And I was wondering if any of you guys wanted to kind of explain what, uh, jim's proposal is for for how we kind of Straighten this out, but yeah, what's what is the problem if somebody has your mnemonic? Right, so basically if you use to the east the the same amount to generate the validator key and withdrawal key withdrawal credentials Which is most home stakers users, but it's Much less often happening for big operations Where it's usually different keys more owned by different people So if it's the same the same mnemonic was used or like what is going to happen Is that now the Um The malicious uh person who compromised the key they can Like you're in a race to who can slash the protocol like slash your validator first Which is if your small validator is not like really a big problem you stop validation you get very small fine Slash slash penalty and it's not that old but withdrawal key compromised means that they can When when the validator exits and withdrawals are implemented in the room and protocol upgrades allows to Either to get back to execution there They you're in a race to understand who gets the money And if you're a small thinker you probably lose the race versus like a Crafty adversary So we we'll uh follow up a bit more about that later on the talk but I wanted to ask a bit about light over silly and specifically the Withdrawal key setup you guys used a genesis. So not the most recent one Could you tell me a bit about like how Lido originally set up with withdrawal keys? Yeah, long story short the initial I think about like 15 or 20 percent of stake that Lido has has been deposited on Threshold signature on on Zero X zero type if I'm being exact for for people who knows basically Shattered credentials that are shared between 11 people Very well known in the industry Who when the time comes when the draw was implemented in rotation of withdrawal credentials implemented They can issue a comment to rotate it though to the Lido smart contract address So, yeah, that was like we recorded it for half a year and We we've had a couple audits on it So like it was like hard to make because at the time there were no smart contracts or like even execution like credentials option for for opposites Thank you for teeing me up for my next question Which is to talk a little bit about where things went wrong with the ox zero ones and there was like basically Well, there's your hot keys and your withdrawal keys and there's like two different ones I wanted to touch on where there were problems. The first one is stake dot us and the hot key issue Does anyone explain what happened? It's quite a while ago. It's only two two years There was 175 validated slash because they were running the hot keys in two different places You might know the details and I keep It's like it's quite quite simple to understand So stake it is a big sophisticated or not operator They they run like I think Maybe billions maybe hundreds of millions of stake a lot of them And they have sophisticated setups. So like one of the they feature they used in in multiple protocols was they had implemented something like like Something that in traditional world would be called like web application firewall So basically they implemented software that ran between the node and the network And it should have catch Code the the double sign events And they implemented for as far as I understand for multiple blockchains including the dream and put out put it up For for the even validation Set up and ran multiple Versions of the nodes with like with the same private key that We're making blocks in parallel and this was usually caught by the their system I'm I'm not like 100 percent sure that because otherwise it could be it could have been that they accidentally deployed it like To things in purple not not like the as the usual setup, but anyway, they did They did have An automated system forliness that at times could have run multiple validators with the same private key which is Usually lead to slasher ball offence in in protocols including the room and They had a system to That should have caught the the problems which is like usually people don't do that like right in in not operator setup because Liveness is not as important as safety slashing for for lack of for for for missing a few blocks Much much much lesser than Slashing like the penalties for for double sign. It's like not worth it usually So that system didn't work and They for about like 40 minutes or like about an hour They had issued blocks in parallel and they were like really lucky that the dream system is designed like a design because like they had I think about 70 million dollars taken or something like that at this time and they only slashed Like 15 validators or like 27. I don't like really like rookie numbers, right? so and that's the case because Not a lot of validators had Had issued the invalid blocks or attestations because It's not like you you don't run the your stake on a single node like in many other systems Otherwise, they would have like five percent slash or like 15 percent slash which is like much much higher and Only after that they Monitoring kind of learning system had triggers. So like they're they're in this With this problem for a long time. So there had been some problems in there as well I think initially described the setup as sophisticated. I think there was a very kind way to refer to it I would I would call it dangerously over complicated, but I agree. I think that these setups where you're running Multiple instances of the same key in any place. I think you you really like Vasily said you're just prioritizing liveness over Safety and it's I think it's really bad business. It's not it's not good economically It's really bad for your reputation And it's bad for the network as well. So I really couldn't advise against it any stronger Yeah, probably the interesting thing particularly pre-merge I think it's probably a little less likely post merge now But pre-merge you could run validated keys in two places and get away with it for a surprising amount of time Until you proposed a block or until your node was just slightly out of sync Or you just got that little bit unlucky because both validators if you're staying in sync and are testing perfectly They'll create the exact same attestation and that's not slashable. So it's incredibly dangerous And do you think you're okay for quite a while until one day you wake up and you've lost all your money? Just a bit of money actually because of how it didn't work. So yeah, I know but particularly With withdrawals not implemented. It's really must be frustrating to have been slashed Early on in the chain and still be waiting to get your money back out Yeah For sure And then as you mentioned slashing you don't lose all your money But as we touched on earlier if there's problems with withdrawal keys, you can be like really in trouble There was one other very famous issue about two years ago as well by stakehound Does any one remember what happened there with their withdrawal keys? I had to go look this up myself beforehand. Yeah, you might have to give me the background I remember stories and not names Oh, I also know exactly what happened here. Yeah, I read all the pass motors. So yeah Yeah, so The thing is stakehound together with fire blocks Has designed a system not unlike the one that leida designed with threshold signature used for holding keys except it was like Much smaller. I think they only had like three or four signatures out of which two two shots was in possession of stakehound and two shots with possession of fire blocks and Nobody knows what exactly happened because like The the parties have like different interpretation of what happened But what we end like what they end up in situation when the two shares that stakehound was supposed to hold were not hold by stakehound And that could have happened like we don't know why that happened Like maybe it was like problem on the software side. Maybe it was problem stakehound side more communications on like wrong backups system like that's buried in lawsuits. I think right now. So Not sure about that but the end result is All that stake about 72 million by I think 38,000 either. Yeah. Yeah 600 dollars per end or something like that I didn't check if it's operating still or not like if I think it's operating still because Like there can be a miracle. I don't know Someone always transaction rewards now. Yeah. Yeah Some some some kind of like chance that you find like an old mnemonic in the old Jacket or something, right? I don't know But it's essentially unvisuable right now This is how you wind up with your CEO sifting through the dump looking for Quite possibly or yeah going through like any old bit of hardware they can find Um, but yeah, thank you very much, Vasily for giving those descriptions and preventing me from having to do it But yeah, I think we don't necessarily know, you know What happened but what we do know is that if anything happens to your withdrawal key odds They're very high that there's nothing you can do Um, so with that in mind genesis happened in december 2020 and about six months later So there was a change to withdrawal credentials. They introduced the idea of um, what we know is zero x zero one type withdrawal credentials And does anyone explain what they are? Yeah, actually know that one. That's good. Um, so the zero x zero one withdrawal credentials were introduced because we realized That the way we were going to merge and and move forward with ethereum, too Was different to what we first expected. So we set up these zero zero x zero zero keys as bls keys matching what validators have partly with the expectation of All ethereum addresses becoming bls keys and partly just didn't really know what it was going to be so we went with the tech we had um With the plans for the merge keeping the execution layer and keeping the traditional ethereum addresses Uh, we introduced o x o one addresses which are just an ethereum public address So it's a whole bunch of zeros and then the 20 bytes of of a normal ethereum address And the guarantee is that when withdrawals are implemented and Your money comes off the beacon chain. It will be credited to that address um The catch with it is a is a minor one but important depending on how you write your contracts if you you can use any ethereum address um It can be a contract But the evm will not execute when funds are added to that So if you're expecting log events to come out, they won't you won't have that opportunity You need to write the contract in a way that it can kind of just handle being called one day and they're being more money in its account than it previously saw and Dole that out and however it wants from there Sweet and then lido when this came out Very quickly adopted it Could you tell me a bit about you know what that process was like the risk of writing a contract for something that doesn't exist yet? How did that kind of process go and and the kind of migration you might say there's no migration but to swap out Yeah, so it wasn't too hard actually like it's like we have Done a stop that is governance controlled and that stop will be like sometimes like in the future when these roles are implemented It will be upgraded and when the ethereum ossifies enough It will be made un-upgradable So ossifies as well So there is like nothing complicated about this like just a smart contract that that is a stop that does nothing except being upgraded Implementing withdrawals handling withdrawals due to the fact that like you can't know when like you're not getting triggered when either come in Is a is like is harder The ethereum taking protocol is pretty complex and we when you add the liquidity like liquid staking protocol complexity on top of it. It's Um, it's pretty hard research to to do it just right but we'll manage so Sweet sweet. Yeah, uh, you make it sound very easy, but i'm not so sure that uh writing like a proxy smart contract holding I don't know 10 billion dollars is super easy. Is that about right number wise? Like it's not harder than uh writing a contract that calls zero Like right, it's just like the stakes are higher. Yeah, if you get it wrong, it can be a contract that has zero very quickly So at some moment you just become dead inside like when you walk the Easterday, so I used to be a city offer a validator like I had like 12 incidents in a year. So Yeah, um See, there's one other withdrawal credential that isn't necessarily in the spec yet and that's 0x03 Does anyone want to talk about what that uh withdrawal credential is? I don't know what that is I think I think you're probably keen on this but the 0x03 is designed to be a forced withdrawal so at the moment, um You actually wind up needing both your hotkey and your withdrawal key effectively to To get through the full withdrawal process because while your validator is active Your 32 ethys remains locked up. You need to exit your validator and then it can go through the withdrawal process And that's where the withdrawal key comes in If however, you're uh staking so you're not staking yourself You've asked someone else to stake on your behalf and you've given them your hotkeys Knowing that they can't steal your eth because you've got the withdrawal keys. Not a bad setup um You have one little problem in that you can't force them to exit your validator and actually get your eth back And so the 0x03 credentials are intended um to be a way of The withdrawal credentials being able to force exit a validator effectively Um, I don't know if you know the details of how that was planned So like there is no more details because like it's older drafts But I think that 0x x x03 credentials are completely unnecessary. They're just a feature that can be added to 0x1 um Like the same way Handled just much the same way as deposits to be container handled like just like consensus layer processing messages from execution there And it's not even too hard. It's just like needs to be designed and implement test that we are actually working on it right now on design Uh, I don't think it'll we could kind of make uh it into Shanghai. I'm like that's Probably unrealistic, but like the next hunt after Shanghai is like doable I think It the interesting trade-off as a client dev and designing some of these protocols when we're talking about withdrawal processes is Trying to be fair with Who controls which key and therefore what respond what rights they have? um So we've kind of set up this idea that the money will go to whoever has the withdrawal key Um, and the signing key is in control of actual signing and exit And as every time we go and change that you wind up with a whole bunch of people in quite unique situations that were Built around that particular split of responsibilities And when you change the split you've got to kind of make sure that you don't break things for other people Or introduce insecurities because of this particular way they've set things up I personally don't know anyone who built around that like assumption so like Who is actually relying on the fact that validators can exit and withdraw us cannot I I don't know. Uh, yeah, I think that's a fairly reasonable one that You know the the base assumption is that you you are You know, it's your money. So it's your validator and you own all the keys When you start to move away from that into Staking providers and so on you are moving out of the I guess ideal case You know that there is a role for staking providers. I don't mean that as in, you know, this horrible person on my right or anything But you know it does it does The onus is almost on People doing that to to help with the security and manage it Rather than on the protocol itself because it's based around that one key one validator type setup um, so yeah, I think I think it's quite reasonable to make a lot of these changes and and we can kind of push those boundaries A lot of the the actual expectations haven't been really formally and strictly set down either we've kind of given a An expectation in many cases But knowing that we didn't have actually a plan for withdrawals when we'd set most of them We knew that we're going to have to be changers and so there are kind of some really core things Maybe that we've got to keep that, you know, like the money goes to the withdrawal key Not the signing key. It will never be the signing key that gets the money But beyond that. Yeah, there's there's a lot of wiggle room. We do have I I had a similar concern. I did mention my background, but I also like previously ran quite a large validator and ran maybe 10 000 plus validators and At one point when they were discussing this, you know New 0x 0 1 withdrawal type and you know, that would be the way that ultimately like money will come out if you are on the 0x 0 1 you'll need to like swap eventually before you can withdraw There was a moment where that was going to be the hot key That gets to pick where the new 0x 0 1 1 was and I was quite concerned because all of a sudden that meant I was in charge of billions of dollars for a previous a small amount of time and then they were like, okay No, let's make the withdrawal key make that change and I think that's Definitely for the better but on that one I do have a question for the two client devs when it comes to Like I don't say keeping people happy But whenever like people have requests of how to build staking Are you always focused on the home staker? Are you always focused on enterprise liquid staking pool? There's often come into conflict like who are you keeping happy and who are you disappointing? Paul I think adrian expressed a little bit of that sentiment his previous one Um It's it's different for everyone. I think I can't speak for all the client devs. I can't even speak for my entire team I can speak for myself. Um I like to prioritize the home staker. I think I think that's kind of the base instance, if you know what I mean where kind of all other features are Kind of super sets of that And I also think it sticks with the ethereum ethos of decentralization as well um But so I would say that making decisions that put the home staker in an uncomfortable position is probably Almost always out of the question if it's if it's to benefit the institutional staker But I would say the other scenario where if we need to benefit the home staker and it's going to it's going to Cause troubles for the institutional staker. I would probably be more open to doing that just because I have more resources There's fewer of them. Um, so ideally we always want to support everyone We don't want to just like, you know do things to Like to disadvantage institutional sake. It's just because we can but my personal preference is for the for the little guy Yeah, yeah, I think a part of that and almost probably underlying that Is that it tends to be ethereum that comes first And and those core principles. So I as talk this morning was really good in terms of talking about some of those principles behind ethereum and And Decentralization is a key part of that and so that's why I tend to agree with the home staker tends to come first They don't have someone advocating for them and they're the most decentralized form of validation um So yeah, it does mean that that there are sometimes those hard decisions that A lot of people would say, you know, it'd be really nice to stake with less eth Or it'll be really nice to whatever it is and you look at and go But that's really going to hurt the protocol like it's going to cause too much system requirements and Decentralization or it's whatever it is. It's going to reduce security or so on Enable more capture of different things those kind of principles mean that We won't always make anyone happy Because it's actually about making ethereum work well and work well for the long term Yeah, it's to add to that from my experience the only one feature that uh was like for big operators And not for home stakers was uh, actually an advanced key management practices And like we talked about with sigma prime about that. I think in before before idiom I made it to be constrained like start And that was like pretty unambiguous feature Like I I I don't remember like even one case where this interest was he were in in opposite, right? If home stakers can run the note like operate big operators can run the note And all we want is like good practice of key key management And for that the only thing that is needed is a good standard of Remote signer. So like remote signers can be written up Like coded up independently And that's the only feature that I can think of that is like not Usually used by home stakers, but almost always used by Big operators. So that's it I think Yeah, absolutely most of the decisions we have to make are good for everyone and It works out pretty well The external signer is a really interesting one because it wound up with a niche It's being used by a number of systems including damp node to make it easy to swap clients So you put your keys into the external signer and it holds them and your slashing protection And then you can change your validated client and beacon note all you like without ever moving your keys So home stakers have wound up adopting this technology that was really first built for for the big staking providers And that's kind of cool Yeah, the one thing you're describing in terms of where the protocol was changed to favor the little the home stickers versus the like Enterprises was probably with me v boost and blinded beacon blocks Um, this one, you know is one that we're probably seeing a lot of people kind of giving out about now But I think a lot of people don't consider the counterfactual Which was if these were plain text blocks me v boost and might not be like given to home stakers And the fear was that if they were excluded from me v Um, it would like massively impact their award and would massively harm them So, you know, we now have this world where people are giving out about like, you know centralized relay or censoring and that was kind of the trade-off of going for the blinded blocks and stuff But yeah, it's definitely a tricky one And I sometimes I'm grateful that I can just ask tricky questions. I don't have to actually answer them most of time Um, so the next thing I wanted to do a little was look a bit forward and kind of where we're going next in terms of withdrawals Um, so we talked a lot about like their liquid drawl credentials But we haven't really talked much about how it's going to happen You might say and I'm going to try and not ask like when withdrawals, but can I ask um How the withdrawal spec has changed over the last six months We started with kind of a pull-based system and now we're at like push Do adrian pauli do you want to like talk about why they swapped? Uh Yeah, I didn't follow it a whole lot. Um whilst whilst in in the previous pull method But I do know how the push method works now. Um, and I think it is a lot simpler So the method that's going to happen now will not going to happen, but it looks like it's going to happen Um, is a system where the beacon chain Automatically without user input will scan through the validators and look for validators that are fully withdrawable withdrawn And then it'll just pull their eith out and then put it on the the execution chain And it'll also look for validators who have excess balance above 32 f and then just pull that excess balance out and put it on the execution chain So this is um kind of so it's it's it's an automatic process where You don't need to go and withdraw f you just kind of end up with money in the smart contract Which I think is really really nice. I think it's really good for users because It means they're signing less weird messages I know anyone who runs a validator could probably um empathize but The idea of having to go and touch the keys on my validator client or do something with my validator client Apart from just leaving it alone. It's just terrifying and it's horrible. Um, so I think it's really good to be able to just like Have it happen automatically Yeah, there's gonna be I don't know if I So there's there's there's gonna be a complication though and it kind of it goes around these um two types of With drawl keys we have so it's been mentioned. There's a zero x zero zero keys, which is a bls withdrawal key And then which was the like adrian said we did that in the early days because we didn't know what we're gonna do Then there's a zero x zero one where we don't have a bls key. We just have an ethereum address Um, so if you are on a zero x zero one ethereum address Then this automatic withdrawal system is gonna it's just gonna work for you. It's just like when the hard fork happens You're gonna start getting some some fresh heat Um, but if you're with the bls keys, what you're gonna have to do is go and dig up your withdrawal keys from wherever you put it Um, and then you're gonna need to use some sort of tooling um to sign a message Um to say, okay I'm now moving away from this bls key that that i'm proving that I have and i'm switching over to an ethereum address Zero x zero one type and then the automatic withdrawals can kick in Um, so yeah, I think it's a shame that we have to do this. We have to make I would say most users Um go and sign this message Um, but I think it's unavoidable, but then after that, um, it'll all kind of happen automatically so Knowing that that is very much in the pipeline and looking like it's gonna happen where I know lighthouse society implement So I think take it was it lots of the clients is I came on it. Um, I would say if you're creating a new validator today Uh, I would be using zero x zero one. I would be using an eth address. Um, I think It's gonna make your life a lot easier And I think just managing I think managing ethereum addresses is much easier Than managing bls keys just because we've already got trezors and everyone's familiar with it So yeah, one really important point is that once you have switched to an o x o one key So once you've got an ethereum address, whether you started with it or whether you switched through this new system That's it. You can't change it again Um, and there's concerns around that because people want to change it But generally the reason you'd want to change it is because you've gone and lost the key to that or you Typed in the wrong address or something Don't make that mistake. It's like transferring money to that address, right? It's gone There is no way to get it back if you don't control the key Uh, because as we talked about before it's the withdrawal key that has to own the money So if you tell us that oh x zero zero zero zero the null address owns your eth Well, now it does um, and you're not going to get it back and there's really very little we can do about that Well, there's nothing we can do about that For sure. I think um, we're going to see a lot of work and there was actually a talk by a test and just briefly beforehand where it's Now that we can have like solidity contract addresses You can program your like change of ownership in there and not like at with some cryptography or any sort of like multi-seg or anything Too weird Yeah, and that's absolutely a big part of the reason that we've gone with such a simple approach to withdrawals We don't need to do anything fancy on the consensus layer side because you've now got the full power of the evm To to work with things and have upgradeable contracts or split contracts or all kinds of things there That let you have this behavior. So think hard about what address you said is your withdrawal address I would say for most home-stakers it should be something like from your trace or you know your hardware wallet that That is very safe and very secure and simple. Keep it simple. You don't need to do anything fancy and then You know staking providers will use something that is a contract and probably upgradeable So they've got some control over it and then there's a range of people in the middle where you You think about what your needs are and possibly use a very simple contract that could be upgraded But could also just kind of hold the funds and send them to you For sure. Yeah, I think um The you've touched on one of the things that will show up over the next few years And this is the evm having more control over the consensus layer because as you pointed out right now You write an address and money will show up But like no code is going to execute and that's about it. Um, how do you think that's going to change and what is the evm? I don't know consensus layer api going to look like in three years time I'm really hoping it doesn't ever get a say Um, I'm really hoping this stays a one-way thing from the consensus layer pushing to the ethereum layer or to the execution layer Because it's dramatically simpler To get the evm to call out to the consensus layer is a big deal and a really big challenge We've kind of done it with deposits and it's the worst code in any any consensus client is tracking those deposits And we want to simplify it now. We've merged Um, but there isn't a generic system for passing messages from the evm Back to the consensus layer and it's a big deal to try and design it. Yeah, definitely all right, but read hopefully Uh, I hope that the consensus layer can start to push some information about itself into the evm so that you can You know do proofs it to beacon state routes, but I'm with adrian Um, yeah, I wouldn't go going the other way is it would be very painful for sure Yeah, definitely don't want to go the other way But as you said being able to read things about the consensus layer I think it's going to become very important because one of the things that we look at with all of the liquid Staking pools is for the last two years These were two totally separate chains and there was like no information passed even still post merge There is still more or less no information being swapped But all of these pools more or less have oracles to say what's happening over in the consensus layer side And I want to take the chance maybe to talk to facility a little bit about How does exits work for something like Lido because you guys are you know a liquid staking token And you're soon going to be able to offer redemptions for the first time But the redemption queue is very bandwidth constrained So how does that work for Lido trying to allow people to Exchange their st eat for really at some point, you know in the near future. What's that going to look like? Yeah, so, um, we haven't settled on the final design yet. So what I'm talking about is the walking Walk in progress solution, right? It's almost ready though for like presentation and voting in for for the dollar, etc But the gist of it the best solution we can do is to reflect the withdrawal queue in on the execution layer We like try like looked at multiple different approaches like oceans for For the like for the place in the queue Some kind of buffers for withdrawals, etc, etc and it all I think is like strictly inferior to Allowing people to place the like requests to withdraw and we put them in a queue on the Bitcoin chain and withdrawal happens and they get their their withdrawals When they time in a queue passes. So That's probably going to be it's pretty like it's pretty complicated because like there are multiple asynchronous processes happening like the withdrawal requests the Oracle reports about what happens on the big and chain like you said, right We like the needs information what's happening in the big and chain and we can't get in real time It's It's just impossible, right? So Oracle requests slashing and slashing side Like we haven't had any in our lifetime in Lido, but like If it happens, it's like 35 days process at least And somewhere in the middle. It has an uncertain amount of slashing penalty Which no one knows like until it happens how much it is going to be Because it's like depends on the amount of slashing that like are accumulated up to this point So when you combine all this asynchronous process and you need to like give people exactly the amount they Like they they they need to get it's becoming pretty complex protocol But in favor whether it's just the same queue as on consensus layer reflected on execution layer as Like basically vouchers for for the place in the queue Sweet, um, awesome. Yeah, that makes a lot of sense And I think you're not going to be the only one that's going to have a lot of pain with these withdrawal queues very soon because everyone just thinks withdrawals no one thinks withdrawal queue and That actually leads me a bit to the like balance skimming. Um, this is another thing that Jim or like fifth panelist is very like In favor of which is if you don't have any way to take the money above 32 ether that's accumulated All of the big um liquid staking protocols will just start to churn their validators take off the you know Three ether off the top and like set up a new one and the queues will be like perpetually full because you know Why don't you just like go and like kind of skim and then restake? um Can anyone tell me a bit about what you know skimming is going to look like because it's also changed a bit in the last few weeks Yeah, it's a it's a very similar process to to what paul described in terms of finding of scanning through the validators and just It magically happens. It's pushed from the consensus layer The the key thing is that there will be a limit to the number of validators We scan each epoch to to look for whether you've got money over that 32 ether that could be pushed to the execution layer The reason for that is that we've got to fit it into blocks Um, and it's like another ethereum transaction that goes into a block But we're not charging gas for it. Uh, it just happens and so we kind of have to have a rate limiting on it To make sure that we're not effectively just blowing up our block limit and Execution clients are able to process these these blocks in a reasonable amount of time So I think it's something like every two months we get around the validator set But I don't think the numbers are at all set so Yeah, it's Those numbers in particular are going to go through some research and and seeing what the impact is on execution clients of this many or that many deposits But it means that fairly regularly and it'll be in in that order of you know, a low number of months I think that we're aiming for generally Verily regularly whatever you've got over the 32 eth gets swept to the execution layer So you start getting your rewards on top of the transaction rewards you get in when you propose a block now I'm gonna say I think um, I think it's really good because the idea of a validator So a validator would say 33 eth exiting taking an eth putting 32 back in is really bad for consensus clients Because that extra validator slot They they cost us we have to store that validator forever. It's more in ram Yeah, it's really bad So so the ability to like for people to take their earnings without creating a new validator index is really good for the protocol Yeah, there were there were certainly a lot of people gym I think leading the charge being very vocal of you've got to have partial withdrawals before you know at the same time as withdrawals and you know as as core devs we try not to promise things until we're sure but I think all of us knew yeah, that was absolutely right And there was no way that we could could manage the impact on the chain of these churning validators if we didn't have partial withdrawals There is also an economic aspect of it. Uh, if there were no scheming of rewards, uh, then They be corporate like the big state controllers like protocols Exchanges whales, etc. They would get a lot of leg up against small stakers Because they can afford to compound and small stakers can't like it's it's very hard to them so, um, yeah, it's like it's very right move to Make it like scheme rewards a thing The like the the slight drawback of this is Working against the protocols that like staking protocols because you can't really understand If the money you're getting on the overall credentials comes from rewards or comes from unstaking and you need to know that Because like rewards you have to reduce you to stakers and unstaking is like something Like someone is drawing something on wrong or like me a session happened like you you need to walk But it's anyway much better than the alternative Yeah, I'll I'll also echo the it makes much more sense from the compounding thing But if someone trying to write like solidity to keep track of something You now I don't know was this a very bad slashing or was this a skim? So it is a bit tricky like that. Um, I've just one more question before you go to the audience for a few So have a think of anything you want to ask the the panel here, but um When the cappella hard fork comes in and we have this o x zero to o x one upgrade How messy do you think it's going to be? Is it going to be over in two blocks and it's going to be 10 people? Is it going to be 50,000 validators? I I suspect I mean probably the majority of validators on the network BLS credential sort of more need to switch. I don't think there'll be a huge rush Um, I think there'll be you know the initial burst and very likely what will happen is you will submit it and Operation pools in clients will actually be full and it'll probably get dropped from the network And it just won't make it on to chain at all and you'll need to submit it again So we'll kind of have to work through the the usability of that because if there is a big rush It's more than we'll keep in the memory pool basically Um, but I think generally it will be over time because people have got to get their withdrawal keys out It's something you should take your time and do safely and not rush in the first instant Um, and and once it's done it's over. So it's it's fairly fairly one off Yeah, I think we'll have to think a lot about the tooling to easy people as well I know the staking I was talking to Carl from the ef Carl beak. Um, and he was saying that it's likely that the Staking deposit cli tool will have the tooling tool allowed you to do it Um, and I think it'll be a little bit easier because the staking deposit cli now kind of like makes a bunch of And then you need to those um and That's kind of annoying in my opinion And I think what now that the apis have evolved Hopefully what we can do with the staking deposit cli will like you can just input your mnemonic and then Tell it where a beacon node is or even a like a public node and then just have it publish it out to it like that It'll be a little bit scary because you have to touch some crypto stuff But I'm hoping that it'll it'll be okay And I think that I think I'm keen. I know adrian to be keen as well to try and make it nice and smooth for people Yeah, and I would expect there'll be an offline option of that as well so you can generate a file of You know your your switch Command the operation that you can then put on a usb stick take over to a hot thing and and then send to a public beacon node Yeah, you could probably upload it to the file to a web page or something like that Yeah, very likely and it's already pre-signed so it can't be changed. It's not a it's not a risky thing It's just like you Something you move up Yeah, my estimation is about 200,000 validators will like rotate as soon as humanly possible like a safely possible The reason is like I know about Lido we have about 50,000 And every extra second is an extra risk On this on this credential. We don't want to do that like we won't overload the network Of course, but like as soon as humanly possible we do it It's it's very important and About I think like 150 200 thousand of operators are On bls credentials that are managed by exchanges And this exchange that they want to Like to pay their wards as soon as possible again and like converted to liquid stake and derivatives for like custodial like coin based assets cb so That's going to happen also very fast like the The protocols and custodian that Aggregate user stake They will and haven't rotated they will rotate like they will rotate as soon as possible because basically BLS zero x 00 credentials are as useful as a potato The one thing you can do with them is like exit or rotate and like that's just not right Yeah, no, I'm aligned there and I do think everyone will go and upgrade at like a relatively leisurely pace um But the likes of the the Lido's on the old ones and like I will block demons and they're on o x o keys But the withdrawal keys are totally separate to the like hot keys So I don't think there is big a concern But I unfortunately do think and they've been people that have been posting on forums like admitting to this That they have their hot key and they're like validating key on the same mnemonic And they left that on their like validator machine or something similar and that's been compromised And the problem is is you know, I think there's at least 10 if not a bit more jim has probably good data on it but um When capella goes live for those first couple blocks there will be an awful amount of mv on the table for you know, there'll be two people that can