 Good morning, everyone. Welcome to all core devs number 134 couple things on the agenda today. I think by far the biggest one is killed and kind of going through what what happened, making sure we figure out next steps from here. Then, we have some updates on some Shanghai proposals so Alex has some updates on beacon chain withdrawals there's also someone else. I'm sorry, I'm blanking on their name, who left a comment and wanted to discuss something about partial withdrawals. And their pro had a bunch of updates about the IP 4844. And then if we still have time at the end I, I had a proposal about how we can harmonize the query IP process and the executable specs that are being worked on them. So to kick us off. Yeah, Perry, I'll put to you but maybe somebody else in a better position but someone kind of want to walk through high level, what happened through the kiln merge. And then we can probably hear from like the different client themes, kind of specifically about what went on on their side. Sure. I can give a high level overview. So we had the kill and test net launch last the proof of work portion of it launched last week on Wednesday. And we had the proof of stake beacon chain launched last week on Friday, the merge itself happened on Tuesday a bit earlier than expected. We did the merge once by using the terminal override terminal total difficulty override flag, and that was unexpected but that excise seemed to have worked perfectly. All the clients respected it and we noticed no, no weird behavior from anyone. However, once the merge act merge transition actually happened a few clients did have issues with block proposal and or sinking. So let the individual client teams go into detail later on. Since then the network seems stable. I think there are still one or two clients that have some issues, but they all seem to be minor and should be fixed relatively soon. Thanks. One thing I'm curious about the status of the block explorers. I know there were some issues with the block explorers like shortly around the merge and they kind of were were lagging for a while. And can you just give us a quick of it. If you know what what had a mirror. So we, we run a forked version of the beacon chain block explorer, and one of the payloads exactly the execution payload was set to instead of begin. This wasn't an issue in the previous ones because, well, the base fee per gasoline wasn't too high, but apparently in this one or the number of transactions one to high, and in this one it was high so it just overflowed. So pretty much the same thing that tripped up prison that tripped up the explorer, but it's fixed now and the explorer just takes a long time to sync. So I think we're still stuck about lagging by a day or something. Okay, great. I apologize to everyone on the live stream there was a OBS issue, and only my voice was audible so I'll recap what Perry said, 30 seconds. We will also have the zoom transcript which we can, we can add to the notes to get the full version, but basically with kiln launched under proof of work last Wednesday, the proof of state beacon chain went live last Friday. We, there was too much mining on the network. So we have to use it to the override feature in clients which which worked well and allowed us to delay the actual merge on the network which still happened Tuesday. There were a couple issues on the marriage around block production and sinking and we'll dive into into the specific ones from clients right after, but the network was still was still finalizing and is still doing so today. And there was some issues also with the block explorers, some individual overflow issues, and these have been fixed, but the block explorers are still lagging importing all the data. Anyone else have just like general comments on kiln before we kind of dive into the specific client issues. Okay. Go ahead. We'll be discussing, we'll be discussing perhaps after the client issues whether we want to do another dev test that before public test nets given that failure or not. Yes, I think actually we, we might can maybe do that now like I know. Yes, Perry and I and others have talked about shadow forking Gordy x week, basically. So I think the short answer is yes. Yeah. Yeah, I think given the kiln, at least as we believe the specs will be is feature complete and we've kind of solicited a lot of the community to jump in on it. I'd like to just kind of keep that as quote, the public test that right now and continue to ask people to do full, you know application deploys and things like that. I would say, certainly, we should do another transition. And maybe that just called a dev net. And it haven't in life. And I'd also, I'm an advocate for, you know, shadow forking Gordy and or Polia every few days for the next few months. You know, if we can, if we can automate that in a way that kind of just shows us the latest builds continue to work and block production is happening and all that stuff that's going to I think a lot of the concerns. Right. And yeah, I feel I'll be more confident in this in about five minutes but as I understand it right now, none of the issues we found on killed are like spec issues they all seem to be client implementation issues so to me that says like, you know, we don't need like a different public test net, which runs like an updated version of the merge specs we need, as I understand it, like clients to obviously fix the issues that they found. And we obviously want to test that on dev nets. Yeah, unless I think there's like a significant change to spec. It seems like we can just keep killing as is and obviously have clients issue new releases which will work on the network. And then there's a question in the chat about me to boost I don't think it's being used yet. No, and we've reached out to the flash spots teams this week to chat with them about that. So I guess, yeah, to dive into the client stuff a bit more offsets with the first one that comes to mind was that there was a prism guess kind of incompatibility around the encoding of values like the base sheet. I don't know if anyone from prism is on. I'm here carrots. Yes. Yeah, Karen, so you want to give us a quick quick overview of what what happened. Great. Thanks for having me here. So high level summary. It's future layer uses big Indian consistent layer use little Indian. So when we, we have this costume implementation for portable. So whenever we marshal and, and the marshal specific data field, we have to be careful to basically reverse the bike code. And we missed that for the big speed per gas field. And unfortunately, we didn't catch them for the previous test that because, because the basic per gas was quite low and there was a shape people are reporting only the videos are broken so should we fix that first. All right. So we are. Sorry, can you say something. In the chat people are saying the audio. Yeah. I don't know. Sorry, everything's turned on. I'll just I'll upload the zoom recording after. Yeah, yeah, yeah. Okay, okay, I'm going to keep going then so basically the previous test that was unable to catch that because the basic per guess was quite low. So yeah, I'm happy to, I'm happy to. Yeah, I was quite happy to to realize this bug. So as the corrective action, I posted a post molten on Twitter, I'm sure most of you have seen it. But high level summary for that is that we will be up our testing infrastructure. So right now we are working on our differential buzzer for all the API and point, meaning that we will be aiming to be 100% compliant with all the martial and martial and for our end to end testing, we are also adding the transaction generator. And so we can make sure to send all the it's outage transactions to make sure the basic per guess and does not remain low and stuff. So yeah, that's the high level summary. And we're Marius, I understand there was no issue on the get side right it was just the geth prison combination but like geth was working right is that is that correct. Like, we saw this bug because geth prison didn't create any blocks. And we only noticed it because geth prison is such a large majority of the network, and that's where we only saw geth prison but it's probably it was probably also on geth. It was also on geth, pisu and geth at the mind. So it has nothing to do with geth. Got it. Prism, pisu, prism at the mind. Yeah, yeah, that makes sense. Okay, thanks. Thanks for sharing any any questions from people on that. Yeah, I guess in not just the prism, CI, we need to make sure that we have sufficient transaction activity in, you know, kurtosis nightly builds and some of the other testing we have as well. Also, I will say I was looking at the base view on gorelly. And I don't believe it's always above 255. So, we might consider when we do shadow fork to make sure that there's sufficient activity on there as well. Marius, would your transaction buzzer work on gorelly if you have enough. Sure. So, unfortunately, someone stole all of my girly. So, okay, create a new state address. If you send it back. If you're listening to this call and you send it back, then I can, I can test. So, one, one thing that like one thing that I wanted to talk about was that I think we lacked insights a bit into which clients were like missing the slots and which clients were like not proposing blocks or not the testing. And I think we need to up our game there. Like, see the bad client, the odd one out way quicker than we did just on just. Yes, I agree. I know there was an idea. I think it was Perry's idea of like maybe we build like deafness, which have each combination of client as like a super majority client. And those issues that show up kind of the show up in the network stops finalizing. And but I don't know if that would catch. If that would catch the bucket. Is that, is that a suggestion through the nightly builds her. Yeah, exactly. So one variation of the nightly build is just to have all clients together and the second variation is to just have a combination of every client as a super majority. So if a super majority client fails, it's a lot louder. Whereas if, if it's just a minority client, we might not even notice an issue. Yeah, I mean, if that's a tractable thing to do at this point, I would say, yeah, definitely anything like that, that we can do to kind of continuously get a view into things working is really good at this point. So if you can save for the big and blocks for you can tie the proposal into like a name. So if we look at, if we look at a paper for example, if you see the proposal you actually says prison, my house, blah blah blah so we probably can do that for a kiln as well. Yeah, I'll look into that one as well. And one other nice thing that Jim from a test in vouch worked on updating each two. So you can now process a block or an epoch as soon as it comes out and it'll list out all the proposal indexes that have failed or missing attestations or sync committee participation. So no longer have to wait for the explorer to update the explorer will always be a bit slower. But since this tool isn't actively indexing you have to call it. It will be a lot faster and should be easier to debug stuff. Nice. And I think there were also some issues on on kiln between basu and take who if that's right. Yeah, does anyone from either team wants want to give an update. I can, I can speak to that there is a case that we weren't expecting where the terminal block was finalized. And our logic to check that we're descending from a valid terminal block wasn't considering that the block itself those being finalized was the terminal block. So there were some cases where basically we just sit sit there at that TTT. And so we've got that. We've got a PR for that. And also we had some issues with backwards sync, which we have a matching backwards sync PR that is merging today. So we should have 22 dot one dot three snapshot which is what we are going to recommend for using for kiln. But yeah, that's the issue that we were having and encountered and there was a couple of reports of basically sitting at TTT for that reason. And so the fact that it was basically take who was, there was nothing on the take who and right it was, it was just coincidence. Yeah, correct. Okay. Yes. And then I think nevermind also had had an issue or two is that correct. Yeah. So the issue for me that might was that some notes after the transition require restart to make it work. So after the restart, they are working fine. We are still investigating it and to make sure that it was fixed we need to experiment a bit with transition. We found a few potential places but still investigated. The good news is that my restaurant differential father between death and nevermind. So father is sending random requests to execution clients and comparing heads and seems that both clients behave in the same way. And I know, Ergon as well. I think you, you, you knew that Ergon would probably have some issues during the transition but still run it anyway, you want to give a quick update on like yeah what what you learned from this and where you're at right now. So there was one issue in Ergon actually related to Indian as as well. That was fixed. But so we were incorrectly sending invalid block hash for a valid block. I think tackle was incorrectly ignoring our incorrect invalid block hash, and it was skipping like resending the same block. Though we sent invalid block hash. Also, because Ergon was quite late to the party. I personally would like more time to, we are still refactoring our sink sink code. And given that killing like quite a few issues discovered during killing though maybe not like theoretically blocking but still I would suggest not to rush the merge. Take, take it slowly spend more time on testing more like much transitions and things like that. And I personally would like to spend more time understanding how sink works on the consensus layer side, the difference between optimistic non optimistic sink performance implications. I thought it test because I'm worried like what we were discussing in the chat really recently is that what the kind of the performance implications so for like consensus layer sending us when you have to sing something and then consensus layer sends, sends block, every block to the execution layer is it okay is it not okay. Maybe it is okay but I personally would like to spend some more time thinking about it and also testing it. Thanks for sharing other other client teams, I think those were all the ones that kind of had issues on on kiln specifically, but did I did I miss anyone. Okay, I guess not. Yeah, so in terms of next steps from here like obviously I think it's clear to everyone that like we need more testing infrastructure and things like like running shadow forks of Gordy and rerunning through the transition. A couple more times. I think in terms of like, you know, rushing the merge or not and timelines. We, we probably have another month or so before we need to make a call about whether we want to move to test nuts or whether, you know, whether we're not ready for that. So I feel like that the next step is probably to spend, obviously the next two weeks and then possibly the next four weeks, improving the testing infrastructure, finding these these these these issues. And, you know, growing confidence in our, in our implementations, and then we can probably make a call about, you know, do we feel comfortable moving this through the test nets or not. And if not, then I think at that point it's like we might have to discuss potentially pushing back the difficulty bomb. But yeah, I did think we probably still have like one month basically until until we have to make that call at a high level I think that the bomb is going to start being, you know, noticed early June, around mid June, mid June to mid July we probably have like 1415 second block times which is high but not unmanageable and late July or the August assuming like things are the same you probably are looking at like 17 or more. And that starts to be, you know, much, much greater delays and just considering the time it takes to generally go from test that's the main net. Yeah, I think we basically have like, about a month where, or where we can kind of grow our confidence in these implementations and then have to make a call. And yeah, I think, you know, the more iteration cycles we get of testing of shadow fork in that period, the better. So, one, one, like one thing that I'm thinking about is like, I don't think we, like, I think we uncover a lot of facts. And, but we don't like we don't really notice them we don't really recognize them. So, like, there was this sync aggregation attestation thingy whatever we're like that that was like at like 60% or something. And then, after two weeks, someone decided to look into it and it turned out that another man was prison was just not sending sending some something. And I think that is like that is a bigger issue like I think we do trigger a lot of bugs, and I'm pretty sure that we trigger the the prism, the prism base fee encoding back at least five times already. But we, but we never recognized it as such so I think we should spend more time building infrastructure to recognize these bugs and I would like really urge all the client teams that if they see something funny on or something interesting on all on a test net or wherever then they should reach out to the clients that they think are affected and then and like really look into it, instead of just saying okay, there was there was pretty funny. But if I restart my client then it's away so it's like a non issue. And I think we do that way too often. I think we could probably come up with something like key 1010 or so p indicators that a test that's healthy. Right. And it's not, it's not just finale finale is good. You know that's what you hope to see always on main net even if there's some sort of issue with a client here there. But there's other things like no one's looking at the sync aggregate thing because no one's really running. And so it doesn't really matter and just kind of falls into the way side but that's an indicator that something's not right. Right. So there's that there's the number of blocks per epoch it should be 32 almost all the time. There's the number you know there's the the finality there's the amount of the percentage of attestations actually making on etc. And so I would say most of our most of our monitoring and most of our kind of like integration testing should be looking at a number of these things rather than just finality which I think we rely a bit too much on the two thirds metric there we should obviously can let a lot of air through. I think we should also think more about test cases for high because Mario is doing great work with this test and I think we should all try to add more test cases here. Yeah, I mean I'm definitely in the mind that at this point if there's one or a half resource from each team that can work on testing DevOps and other things that we would be in a much better place come four weeks from now. Yes, agreed and I know that I don't think he's on the call but Frederick from the EF has been trying to gather people from different teams that coordinate all of that so. Hopefully like we, we can just have more people from each team but also kind of be a bit more proactive and sharing the stuff that's being worked on so that everybody's aware of what everybody else is working on Frederick is here. Yeah, yeah exactly. Yeah, I sent that out. So, and I sent message to the client teams as well about it. So hopefully that will get moving. Maybe just to touch back on one thing that Andrew brought up the thing around like sinking and and box sending the blocks of the execution layer. I'm curious check. Generally how people feel about that like, is that something that can realistically be changed before it emerges that something that like we might want to improve shortly after. So it's something that can be lever like the execution layer can already leverage the information do whatever it wants here, the, the consensus layer. If it's sinking and sending you bulk blocks means that it is not at the head and it literally doesn't have a good piece of information for you to decide how to like do reverse sink in any sophisticated methods from the head. It's walking its way forward. And that's, that's just the case and so you can either execute as it does that, or because you know the current time in the world, you know that the consensus layer is not quite actually at the head. You can wait until the consensus layer gets to the head, and then do whatever sink techniques that you like to do at that point. But there's kind of a bit of a chicken egg problem here you can, you can lock step sync with them, or you can wait until they get to the head and then do whatever thing that you want. And there's a sufficient information to be able to make either of those decisions. That makes sense. Does anyone have any help to talk about that more and some of the channels and stuff. If people want to discuss different design decisions. Yeah, I think yeah it's been discussed a bunch so we can we can keep kind of the conversation there. And was there anything else that people wanted to discuss about kiln or the merger just like testing specifically. I would like to quickly discuss safe unsafe and finalize tags. So, yeah, there is a PR opened into the execution API as repo that just that finalized block tag to be there in Jason RPC. And on the last call we roughly decided not to have safe or was a bit uncertain about that and one of the suggestions, one of the one of the things that we may do for safe block tag is to. Is to use justified justified block for it as a stop gap, until we get safe rule implemented, like in in its like full proposal that's made by then credit and the detail. So this using justified block is pretty cheap from CL standpoint, so it's just responding with just sending these justified block cash to EL, and it also brings a safe block closer to the head than the finalized one and it's a truly safe block. It's not going to be reworked as human that there is the honest majority and the synchronicity. Also, yeah, but yeah it's it's still not that close to the head. Like, as we previously discussed, as safe could be. So that's just the proposal and just curious what people to think about it. I think that's good. I think that that that is the block that we that we should declare safe and Yeah, I think it's nice to because it gives the exchanges and stuff of chance to begin using this and the algorithm can improve. So are you here. Can you chime in on how you feel about just by being the safe current. I mean there's definitely no downside to using that for now, I think. Yeah. Yeah, I mean as an update on that is this like, it's unfortunately turned out that it is much harder than we thought to define a safe head with LMD. So, yeah, I sense so optimistic that we can do it but but yeah it's definitely a good idea to have this intermediate solution. I would say there's no downside the downside is that we can't point latest at that because too far behind. Yeah, yeah sure sure it's of course but it's it's safe, right that justified so like in the in the safe that sort of framework the justified one is definitely safe. Do we do is there a reason we don't have a justified tag. If that's, is that a useful thing beyond the safe tag, like, can we imagine execution or people will talk in the RPC wanting to know the justified. It might, it might be if we had safe justified and finalized we're safe and justified or just kind of equivalent at this point that does give an additional granularity of progressive confirmation in a sense. We're not finalized but now the assumption on this break this changing is much higher than just safe. But you're also kind of just giving users more choice which may or may not be good. So yeah, I'm not opposed to exposing justified it, you can definitely think of these cases where it's nice. Exposing it seems like the right thing to do and we can always just in the docs make it clear that hey you should probably just use safe. If you if you don't, if you don't know what you're doing you safe, but we also offer justified and finalized or whatever I think but I feel like we can solve the problem of the too many choices be a good documentation. And the issue with that is, once we have safe and justified things, then, like, safe, less safe than justified, I think that's already already weird that safe is less safe than finalized. Marius, it sounds like you're being swallowed by a storm. You told me I could, you told me I could go for a walk. But we got the point, we got the point. The semantics of like safe versus finalized justified is weird that safe is less safe than those two things. You can just call it unsafe. I would not say that it's like less safe. It's safe on the finalized and safe. Yeah, I mean finalized, justified and safe. All three are safe on the different assumptions. That's how it should be better. I'm open to other terms for safe if people have them. I'm also fine with safe. Confirmed is a good alternative, but it doesn't convey the same meaning. I think confirmed is very dangerous because it has the proof of work connotation. And so people might confuse that with, like, literally finalize. Or the last, when is the block confirmed six times. Yeah, exactly. Yeah, something like that. Okay, so finalized, justified, safe, unsafe, latest, safe to justify it, latest to unsafe. Do we want to cap this all? Is there, yeah. Is there a reason to have unsafe and not just keep late this. I think, I think they're the value of trying to over time rename latest on safe helps inform new developers into the ecosystem that they should not be using this thing. Especially once we have better safes, something more close to head than justified. And so I think getting that name change now is the best opportunity to get the name change we're introducing a bunch of other terms. And maybe in, you know, three to five years we can definitely deprecate latest. But I do think there's value in making it very clear that you should not be using this unless you really know what you're doing. Okay, so I'll just probably submit a PR or update the existing one and we can proceed with that. Okay. To recap, you will so finalized and then justified the safe. And that's it. Yeah. And map. I guess inside of the consensus layer spec is where we probably map. We just do the safe algorithm is returning justified for now. And then we can update that algorithm future. Yeah. It will be probably sometime difficult to explain the difference between finalized and justified for end users. I'm not sure if it's going to be that much useful, but anyway, having the granularity is always good. So, okay, let's have this all. Also, like a minor question is what should yell respond with when once it gets finalized requests before the merge. And I think, yeah. And finalize and justify it and save before the merge I think it should respond with error, which is, which will allow to avoid any bugs or unexpected things happening if your request and finalized before the merge and got something meaningful other than error. So, yeah, I think error is preferable option, unless someone has any other opinion. I think the Genesis block is more likely to lead to weird errors. But I agree. I just I would worry that someone has some sort of setup, where they're trying to switch from confirmations to finalized, and then all of a sudden they go from thinking something 10 blocks ago was equivalently finalized to nothing ever in the chain ever being finalized and I worry about edge cases there I think an errors. The other option would be for execution clients to have some sort of CLI flag, they can pass to say, how far back do I want pre merge finalized to be so that you can just define finalize to mean latest minus 10 or whatever. You can always do that but maybe it should be part of the spec. It seems a bit. Yeah, I mean, their software that's deciding if something's finalized should maybe use that rather than putting it inside of the client. Okay, so it seems like there's no real objections against an error, and it just harmonizes the behavior across everything. Is it possible for just RPC clients to find out when the merge is going to happen or when it has happened. They can ask for the difficulty or pre Randall. And if that exceeds six four bits than the merge seven. That value. That's I think. I'm thinking of that developers who want to be merged ready and you want to deploy your app before the merge happens and you want your app to smoothly transition to pre merge behavior to post merge behavior with regards to finalize to be safe yada yada yada. I'm trying to think like, do we have a good story to tell them or a good narrative for how they should build their apps. What should they do, you know, should should they be checking difficulty equals zero is that the right thing or can we give them like an actual is is proof of stake versus is proof of work query that they can run. And maybe the difficulty thing is perfectly fine. I don't know. We can use this error in response to finalize finalized requests because the marriage. Yeah. Yeah, yeah, yeah, I know it's like no code flow on errors. Yeah, yeah, I agree. I agree with that. But difficulty zero would mean that the transition is has just started and is in progress. I mean the first block was difficult to do. And the merge is finished is considered as finished once this first this transition block is finalized. Right, right, so they can actually use finalize until sometimes sufficiently after the merge, which is long after difficulty has switched to rando. Right. And I mean sometime being like 12 minutes or something. Okay. Anything else on Jason IPC. Sorry, I had myself muted and I had some closing comments. I want to give users a a more clear way to like a reasonable way to find out when it's say it's I use words safe when it's a reasonable time to switch to using finalized that doesn't involve them like querying things that are airing and then having to set up that alters their code flow. Maybe we can give them a simple json rcpc method or something they can query that says is now the time like is the merge fully complete or is finalized available yet or something along this line just adaptive developers don't have to put in these horrible hacks just to build good apps around the merge. Yeah, there might be a good blog post to, yeah, we get some of the stuff in like, yeah, about these new tags and also talk about some of the ways they can use them and, and maybe some of the logic you can use to kind of assess with the merge happened and that kind of stuff. And we can definitely organize calls like with application developers and, and, and walking through it and also just get their concerns about specific flows. If we don't provide and like this graceful method call that will return that the merge has happened, they will have to rely on errors before the merge and yeah, absence of errors and gets finalized block as the signal that the merge has happened. I mean that's from json rcpc it will not be possible to get. I mean you can look at the block header right. Look at the block header and that's what yeah because some stuff will be zero after the merge. That's during the transition. So you'll you'll know you're in the transition but you won't know the merge is complete. And that's when you don't want to switch over your strategy and your app until after the merge is complete. So right now there's no way for other than just like trying things and getting errors and then like catching the air and changing your behavior based on getting an error. There's no way for app developer to build something that changes behavior once the merge is complete. I'll go I'll continue the supply. Yeah, or we can create a smart contract which just has the difficulty outcode in it. And then people can just query that smart contract. That doesn't work. So, you want to know when the first block was finalized and you cannot query for that. Yeah, without the first five years. You can implement this. This is like a five line change. I just don't really like the notion of having a new JSON RPC call for like 12 minutes, or that that is important for like, I don't know, maybe maybe like three weeks. And then no one's really want to use it or need needs that anymore. We can use it again when we change the consensus engine again. And then for writing some pseudocode to show how people can decide this from the, and then libraries can write a function if they want, you know what 3js. Yeah. Well, we can take this. Yeah, we can take the software. Okay. Next up, I guess, yeah, before we move to the next thing, anything else on the merge itself or kill or testing. So, Alex has an update about be concerned with draws, and we also have someone else I'm sorry I'm searching on the zoom screens but there's literally too many people. We had someone from the Lido team who had a proposal for partial withdrawals as well. So maybe Alex if you want to go first kind of give an update on on on on what you've been working on, and then we can have. Yeah, it's the partial withdrawals. Yeah. Sorry about that. Alex you want to give a quick update, and then we'll go to our job. Sure. Yeah, so last time we talked about this. Essentially I think there was a lot of sort of demand for something to organize all different threads. So that turned into a meta spec. I'm just going to share my screen quickly and we'll just run through it. It's pretty short. Sorry. I see. Can you guys see this. Yes, we can. Okay, so yeah, I'm not going to go through this in detail if you want to read it. It's here. But essentially it just like has some pros of like how the draws flow will go and then links to specifications at a high level that consensus layer. And then the schedule is when the draw should happen, and then it puts them into this queue, and then the business layer is also in charge of, again, de-curing the draws into execution blocks. There's a specification for how that works. I think this layer here. There's a PR for the modifications to the engine API because then essentially, again, in some way the consensus layer de-cues these withdrawals they're shoved through the engine API to the execution layer. And what I want to talk about today is essentially two different options here. There's like two different EIPs. One of them we discussed last time in terms of having a new transaction type to represent the withdrawal. There's another option that is essentially some sort of system mobile operation. So I'm more involved, but essentially I was just saying, okay, rather than having a new 2718 style transaction type, we have this new type of thing called an operation. And that's where the withdrawals live. And the reason we want to do that is basically the firewall off, you know, mixing withdrawals from user level transactions. And there's probably some like safety benefits there. But and here's the catch is that one thing we'd really like to have is some sort of logging. So when a withdrawal happens, it'd be really nice if there's some way to just watch the execution layer and know that the withdrawal, you know, has actually occurred. So the point here is that if we go with 4863, the previous EIP that we talked about where it's a new transaction type, then, you know, we can reuse all the existing sort of events infrastructure blogging and AVM. That's great. If we go this other route 4895, which is maybe in some sense, you know, cleaner or elegance. So we basically have to recreate all of the receipts infrastructure, and that basically is a lot of work. So I essentially went and put on this call does anyone have any preferences on either route. Have you had time to look at these EIPs you have feedback. I have a quick question is logging actually very important. So it doesn't have to be. I think it is like pretty nice UX. I mean, there's probably other ways to figure out that your withdrawal is processed as a validator. And yeah, it's really just a question of like what, what kind of facilities do we want to provide validators. Right. I have a comment on that. Sorry. Oh, I was going to say I was the argument against having some way to log this. The argument to not have them. Yes. Right, right. Like, it's complex. If we go I guess the system operations route. Alright, so yeah, I mean I can just click through so basically like to give you a sketch like it's literally just having like, is that important. Okay. I think very weird today. That's okay anyway like basically it's just like adding a whole new field to the block and so because of that we can't necessarily directly reuse, you know the receipts infrastructure that we already have. So we'd have to like duplicate all of that and that's where it starts to get like quite hairy. Right, right and spec says that in case of withdrawal operation, it must never fail. It's like unconditional one so you may basically use these withdrawal operations as logs, or a kind of logs. Yeah, I would. I'm definitely agree that the logging issue, like we should either we go system contract and no logs, or we do something else with logs I am definitely not a fan of trying to get logging in or get receipts in with the system operations stuff. Right, so does anyone. Well, first of all just wanted to say to me kind I think the spec does not say that it must unconditionally succeed. I think it's originally this text that is so if you have defined your, your each one contract recipient, which is something which would never ever accept anything. And we'll never be able to withdraw because no matter how much gas is specified it still play. But I'm kind of so this version, this is just like a balance of it. Right, so that's kind of the both of these options are they both centered around the. Yeah, the gasses, the one where we just do it. Yeah, these are both push. Good. We did analysis of existing also when deployed contracts, none of them rely on code execution and we've spoken with some of the employers that had logs in there and said they don't need. Yeah, I think that the system approach is much cleaner and it's a very important operation so we don't we want to do it in a reliable fashion and transaction is just abusing the notion of transaction because it's not really a transaction. It's just a balance update but it's not like, it's not a balance transfer. And like if we want some logs on the AVM side, things like that then it should be crafted specifically for this operation because it's, yeah, I wouldn't have used the notion of transaction for this. Yeah, which is fine but then I'd probably suggest going with this option to route so we bought out a new operation thing, but then yeah just drop the logs. Because I think it's going to be way too much to like have a whole new like receipts try have to win testing etc to cover all that. And like, do we need access to this from the EVM side, because if we don't like the observability will be there you can observe it by the looking at the header, and there was draws as was mentioned, they cannot fail right so if you see there was draws in the header, then you see all of them. But the question is whether you need this observability from the EVM side as an EVM opcode. Right, and I think I think part of where this came from as we looked at the existing zero, sort of each one credentials that have been deployed and the only real thing anyone was doing was logging. But yeah like Danny said we've talked I think to some of the players and it's been fine. You know, Rocketpool is really the only one doing the logging and they said they hadn't actually expected code execution but just kind of put that in there as the best coding practice. So, you know they're not even necessarily expecting more. So, So, yeah, and I'm sorry I'm not really up to speed and all the details. But I agree that for them as transaction is abusing transaction because of all transactions. However, block body is now a list of uncles. It's going to be empty list of ankles after the merge, and it's a list of transactions. There is a lot of code out there that patches bodies based on this and is protocol, which have the capacity to request these pieces of information from another peer. If we want to add a third aspects to block body. I think that might be a lot of work that a lot of code needs to be rewritten. I would like to have heard Felix or Peter have any thoughts about this. I don't know if they're on the call though. I don't see them on the call. So the, the two options for where to put these student transactions on block I think right now are just append to the end, or take over uncles previously Peter had argued pretty strongly against taking over uncles and he strongly prefers depending new things to the end and just throwing away like accepting the cost of the extra bytes. Will there be a lot of withdrawals? Can we simply put them in the header rather than the block body? We can borrow them and be a lot of them. A fixed amount per slot and they will be. So we could decide on a number and you know it affects kind of some of the UX here. But the execute is already bound to approximately four ish per slot. So after you clear out maybe some large amount of withdrawals in the beginning. The bound doesn't need to be much more than that. And the consensus layer will have a bound on what how much putting in here because there will be a maximum cost of this operation on the system. And that number can be. It seems and please correct me if I'm wrong here. It seems like we have rough consensus around the system level operations approach and it's like a question of how rather than like if. Does that generally make sense? Yeah, and no logging. Right and no logging and I think so. Alex, I don't know. I think the ether scan people had said they were fine with like either option as well. Like I think the one thing we want to make sure if there's no longer news yet that folks who monitor this stuff are still able to to access it and expose it. Right. And if you're going block by block like the withdrawal will be there and we know that it succeeds. Right. I think the one use case you don't really get is like, I'm a validator. I turn on my node and I just want to I have, I know my withdrawal index, and I want to ask if it happened or not, you know, where it happened. Right. Otherwise, yeah, you can scan. I mean, and they're sequential. So you can do a binary search to find where you're where your receipt happened or actually yeah that you can actually know if your receipt happened very quickly, because you can look at the latest withdrawal. And if it's greater than your receipt index and it has happened. So there's there's there are things that you can do without logs to probably handle most of these cases you care about outside of the. Okay. And Scott, you have a hand up. Yeah, just briefly wanted to ask and once maybe this answer the document, but potentially putting the full withdrawal into the header. Like what's the size of a withdrawal just the two address and the amount or is there anything else, because then it's basically barely bigger than just putting the hash in there right. It's an it's an address, it's an amount and currently an index. And what's the index is for and the executioner. Differentiate. Yeah, it's to differentiate it's when we were going with the transaction method. And any sort of logging and that kind of stuff it allows you to differentiate. It also would allow you to do, you know, search, like I just talked about more easily. Not necessarily amount amount and address you're not necessarily unique, given partial goals. Yeah, but then that still seems only two extra basic exercise of just the actual maybe it's just the easiest. But you would have more than one per block. I guess. Okay. In terms of next step does it make sense to move the system level version so 4895 to consider for inclusion and have people kind of keep obviously looking into that and and figuring out the course around it but just so Yeah, we can make sure we're kind of all focused on the same thing. Sounds good to me. Yeah, I guess yeah does anyone have an objection to that. Okay. So on the on the partial withdrawals, I just wanted to say, yeah, I have this tracking issue on the consensus layer. There are three key features here one is to fully withdraw ex validators. The other is to change credentials from the LS credentials to execution layer credentials, such that you can perform withdrawals and the third is a partial withdrawal option, which I'm working here right now. I think that this is one from the from the perspective of the execution layer. All of them just look like things being withdrawals being dequeued into the execution layer and so that that complexity doesn't really matter but from the perspective of validators and features I think it's a pretty critical feature to not put crazy pressure on validators exiting and validate the term. So it is, it is, and has been kind of in the consensus right now. Right, and RGM do you want to take maybe a minute to kind of explain what your proposal was. Yeah, it's kind of obsolete now, like the new push based proposal is definitely more preferable from the point of view of liquid taking protocol as well. Yeah, I just like to know that partial withdrawals are crucial for us. Yeah, and I think for even just for many other use cases as well they're pretty critical to the experience and Yeah, just for the health of the beacon chain you don't want people having to withdraw they're like 33 that's something east and then just read it positive on the other side. Yeah, and we've got a second part of our proposal which is more related to the question of how to pass some intentions from the execution layer to consensus layer. Like when I validate I'd like to rotate its keys or do a forced exit, but it's out of the scope for this call. Right this seems like more of a consensus layer call type discussion. Yeah, and I, I'm happy to have had a chance to read the proposal but I do think that that is a very nice feature and actually protects against a couple of like weird withholding attacks that we should talk about. Anything else on withdrawals. And so sorry and the maximum is four or 40. The four is the amount of number of exits per slot currently. And so the number of withdrawals per slot you would probably have in that same order. It depends on the partial withdrawal scheme, but I would say four to 16 or the realm of what we do here. There's no execution block in a slot do those withdrawals designated for that slot can move to the next one she had eight in the following. Not currently no. That would put a, you know, unbound costs on the execution layer. But if it's only four or even 16 maybe we can put them into the header. But if you foresee that in the future it will be more than maybe to be future proof then yeah it should be better to be to be in the body. Right. Yeah, and we yeah I think we can take that offline as we're working into the implementation details. Anything else on withdrawals. I could also say that they're from the point of view of taking protocol, there might be a slight desire to be able to distinguish partial and full withdrawals like maybe different addresses but it seems to be too difficult to implement not not very crucial for us. Just to note. Right. Yeah I mean with the beacon root opcode. All of that becomes possible. It's a matter of that. Okay yeah just to move on because we only have 20 minutes and at least one big topic less. We have had an update on the IP for for eight for four. They were pronounced that right for the for the sharp Bob transactions. Exactly. Hello everyone. If you're very quick updates. So, since last all cord of school, we have worked on the consensus packs execution API is. We've got a matter spec to link everything together. I think that relates to the proposal Tim is going to share where we're trying to find structure for this cross layer. Yeah P process. For now we'll just try and work with every specs repository out there. We also have execution API is now and we have benchmarks as well as implementer notes. I'm not sure if everyone had the time to look at those guns, but it boils down to having to batch verify these blob transactions to work around the processing time issues. This requires some minor reflect ring to take multiple transactions and verify them together. And then the blobs within a transaction can also be batched verified. On all I think it's not too bad. And the same applies here to the consensus layer that can also batch verify blobs. And we're working on testing with prism and work on tooling to, to try and get the definite running. Nice. So, if you have an idea they are on how to say someone sends me a batch of 1000 blob transactions every single one of them isn't valid, but they're constructed to be as hard to as possible to verify. We're talking about them. Well, if you receive all those transactions from the same pair, you can batch verify them, and then if everything is invalid, you can start scoring down the pair. One of them is valid and you need one transaction to go through. And then you have more of an area situation, because once you figure out that the batch is invalid, you'll have to go and bisect the, the thing to find the valid blob transactions. In the end of 3D you, you want to like penalize pairs that are giving you this bad information in the first place. Like this is like provably bad. So you, like, it's objective. And it doesn't affect consensus like this is in a transaction pool, like this is the step before that. So I think we need to improve the, the peer scoring system in the exclusion layer to help filter out things when things do go wrong. I can speak for all clients. Yes, do not do peer scoring. And there's no such concept. The consensus layer has extensive peer scoring, and it really does help avoid problems with like bad attestations or these other high throughput things. What about the years right. So we don't even ban pairs. No, I mean, no IDs are free. So there's no point to ban peers. We do kind of have some IP limits. But yeah, we don't spend a lot of time scoring peers, we kick out here if it does something bad. But no IDs are free. So I guess the point is, you can get 1000 these transactions from somebody that on the first you're going to stop processing. I mean, I can set up 1000 node IDs, connect them to a guy and send them 1000 transactions from different identities for free. Well, there's some processing power because we need to generate these transaction at some point. And then we use them. I mean, the people that I sell them to will not remember them indefinitely. If I have enough of them. And so I could just spam the network with them. And what I'm curious about is, yeah, how costly it is to avoid the attack. A single batch of blobs is somewhere between 50 and 70 milliseconds to verify. This batch can be very large. I'd say it like the 40 or 50 inch milliseconds is for like a single transaction. Like the batch versus a regular like single transaction is not that different. So batching is important here where we like avoid cost overhead of adding a lot more to verify. In the end, like it is processing and we should just like use that information that if you're getting something that's wrong to try and ignore more of that before we get to deny all of service. No different current transactions. So like sections are invalid. You all also ignore them. And hopefully, like, you cannot deny. You cannot toss a Gav node, but just keep by keeping this, this lower the smaller transactions up. It doesn't matter if you send like a thousand small transactions or like one large transaction if the verification is not does not. Like, if the PR that sends you those is not held accountable. You do need some system to give feedback to what is being received. Right, but like the only tool that is that we can trust is that if someone sends me something, then I can disconnect from him. That is the only guarantee we have. There is no guarantee that you can rely on a scoring or to be effective effective. Right. And you're saying no IDs are free, but IPs are quite as free. Yes. And yeah, so it might be possible to ban IPs, but that can be tricky. It seems to work for Eve to like the whole pair scoring system. I think it's not too difficult to emulate some of that small substance enough to say like if transaction is invalid. We should consider this pair as as bad and prioritize others. Yeah, with it, I mean, I guess they do have pure scoring. It's just a kick, you know, that's binary. This is binary to if the transaction is in fellow that's all because of the pair that's sending you the transaction. It's very clear that there must be anything. Yeah, yeah. I just don't really see. Yeah. Okay, we don't have to go too deep into this actually on this call. I'll read up on the benchmarks and specs. Yeah. Oh, sorry. Go ahead, Proto. The agenda, as well as the metal spec links the benchmarks and these optimizations. Right. And what I was going to say is just that I think it's at least worth considering as well that there's no necessity to have a memful memful support for these connections live right at the time they could remain that. So basically, this would not necessarily have to be a binding constraint. Bring it to my net we could block without support and then, for one, of course, it will be slow wrap up for using those and it's fine for me that at the beginning to just only because of course notes would support having that locally fetch the memful so like you, they could just run their memful so they could operate and they could break it off. Like a separate p2p network for that like basically, it's fine if we only add that support level later, of course, not ideal, but so this is not necessarily like strange to the time. Right. Andrew, you also had your hand up and I think we could down and brought it back up so you have any comments you wanted to make. Okay. Anyone else have comments thoughts on 4844. I do think that benchmarks from a number of or stress test from a number of different places are probably pretty important. You know just the consensus layer decrypting one to two megabyte blocks, you know, and then passing the execution layer that there's just going from 20 kilobyte blocks 90 as we are with the merge I don't expect things to burst the seams, but I do, I do think that it's not necessarily that once you get to one megabyte that like little things we didn't expect start to operate in different ways that we expected it that was, I don't think any of this is going to be intractable. Solvably, but I think that it's good to investigate so sweet. Anything else. Okay. I'm last on the agenda. I had something so we discussed this. I think on the last call. It's not on the discord work after. Basically, the, the core IP process is kind of reaching its limits with the merge where we have a completely different process on the beacon chain then then on mainnet. And we are starting to have proposals which clearly span across both so the two things we talked today are very talked about today are good examples of that. It's quite hard to like reason about like what the entire spec for something should be and and how the different parts all work together. In parallel, there are folks working on an executable spec for the execution layer, which aims to kind of over time complement to replace the yellow paper as a clinical spec for Ethereum. And so I had a proposal that I put together about how we could harmonize all of this. Just shared it in the chat, and a very high level. The idea is that we would keep core EIPs as the way to like describe changes provide the motivation to rationale. And also just have like EIP number that's easy to reference within the community. Use these for both consensus layer and execution layer changes. But then, over time, basically move the implementation sections to the execution specifications, rather than having them live directly in the IP itself. And so that you know the benefits we get there is that a is like harmonized across the beacon chain and the execution layer be you can link both so if you have any IP like you can change withdrawals. You can just say hey here's the change the execution spec here's the change the consensus specs and maybe even the API repositories and then see there's always been like this big concern with like that. We don't have a lot of EIP editors. So we wanted to be easy for them to actually review EIPs. And one of the, one of the things that's actually quite hard for them to review is when people put links in the IPs because there's a bunch of dead links over time it's hard to assess the quality. So by having links out to just the different specs repo, you can have a pretty easy to enforce rule that's like you only allow links that you know these two or three repositories. And just block in the IP if it has anything elsewhere. And then if you know the IP author wants to add a whole bunch of links as part of their PR to the specs repo then they can do that. But it's not it's not like blocked in the IP process. I know Greg you had some comments about this Greg still on the call. Yes, yes. So yeah Greg you had some comments about this. I'll let you share them. I also put together an East magicians link for people to discuss. Yeah, great. A lot of this will just need to discuss as editors, we've only got about seven minutes left so I don't think we can dig very deep. There's some good ideas there but I think it's a lot more intrusion on the EIP process than we want to see. And in some ways it's making it harder. The whole point of the executable spec is is another client. So, in the usual process the clients, often with the help of the IP author implement the IP, and the beauty of the executable spec is that once that client is running and is on the main net and in consensus, that client becomes the reference. So I actually don't expect that a core IP, the IP could be a totally complete and accurate reference when it's done. The, the network itself is ground truth. And so having one client that we can point to and say we intend for that to be the actual reference is great. But whether we try to pull that back into the IP as a diff against a particular implementation doesn't doesn't really seem to help matters. I don't think that's where the bottleneck is. And I don't think the issue of references is really directly related that that's a different discussion we're having. I disagree on that one too, but and then you also have your hand up. Yeah, so just a quick follow up on that and then I have a quick comment on the consensus layer. It isn't a full client. It is actually, you know, implementation of core state transition logic, such in very non optimized ways in ways that just expose what the logic should be rather than the implementation of logic as it will be in a client and so it can't run on me. It also doesn't have network interfaces and other things. So, and then we can build test sectors of it so I just, there's, there's a different there's a spectrum of what you can do with it. And I just wanted to say that a lot. And then but I will say, I don't know if this helps our EIP editor problem. I mean, it just shifts the burden to a different highly specialized group. Like consensus layer there's a handful of people that have the ability to review these types of specs and provide rich dynamic feedback. And sometimes there are PRs that are open for a very long time because it's hard to take the time to to dig into it so I it maybe is useful and getting more people at the table but I don't know if it like solves the editor problem. I do think it maybe solves other types of problems. I think it makes it harder for editors. I can't. We have to become experts in yet another thing. To be clear, right. Yeah, it's not expected that the editors would have to review the executable spec as well. Like I think like, so like daddy said you do have different people who then review the actual spec and like they might be the same humans and pose that. So you don't impose basically like a technical review of the executable spec before merging the EIP itself. Yeah, so so EIP editors would only be responsible for ensuring that the format is correct that like, like the tooling says the right things, but the actual review of the content of the diff would be probably by, you know, core devs. Right. So, I guess I don't see how this helps. I mean there's a being a huge overlap to like if I, I would have a hard time reading a metaspec and being like, Okay, this sounds fine. I'm not going to go with that. That actually has all the logic. I just take a look at the wasm spec. It's entirely declarative way back in appendix there's an algorithm, and they make clear the algorithms not the spec it's an example. The spec is totally declarative. It's up to you how to implement it. So there's, it's just not clear that this language is going to be the best way to actually say this is what I want to do. I'm going to ask someone with an idea to improve the protocol to say, Oh, but first you have to figure out how to say it in this specialized language that may or may not be the best way to express your idea. You know, a paragraph of English can turn into a page or two of code when the English was totally clear to anyone who knew how to write the code. Your hand. Oh, sorry. Go ahead. I was gonna say, I think that's a fair criticism. I mean, I think one of the it's kind of a trade off right like a lot of things will be easier to express without having to describe the current state so that that's a problem that comes up in the IPs a lot today is that to say how you're changing something you first how to have to define how that something works. And you know that this code diff process will improve that part of it but you're right there are some changes that are going to be way easier to describe as English that you'll have to implement as Python. If we go down this road. So, yeah, I know just enough Python to hack a script together, or to to read fairly simple descriptions of structures and stuff, but if you do take a look at the consensus layer specs that's that is the requisite knowledge. It doesn't use anything Pythonic, and it does not compose things in weird ways and is pretty much. It doesn't use even like complex. Not complex which is Pythonic type construction such that you know you have for loops, you have variable assignments, and you have very simple data structures. I think there's, I'm not trying to claim one way or the other on on the best of use. The IP process close out, and then this this this executable spec is its own thing. They'll have to be some way of vectoring over, but trying to do both in one process I think will only make it harder. I mean, when I put in code, for instance, that code is often going to be go, not Python, because I might have implemented the algorithm in gas, and then I can put in code I've actually tested. And translating the code to Python isn't that hard for the core group. It's doing it, but it's hard for me. Now, if, if we want to pull that text back into the EIP at the end of the process that's not too hard. Or if someone who knows that stuff wants to become a co author. That's fine too. But yeah, Martin you were about to say something. Yeah, well initially I was going to say the same thing that certainly doesn't need to be a full blown, you know, a network machine that can live on maintenance. But then I want to say that, you know, I've spent quite a lot of time reviewing it and also implementing it. And the English language is great. But when you translate the English language into actual code when you that's when you find all the corner cases and things that were implicit in the English language and when you put into code it needs to you need to make it explicit. And that's when you find the quirks and the corner cases. Which shows that this specification was under specified was too big. So I think it's good if we get closer to how it looks on the outside where it is code where the author who put this down was actually forced to figure all these things out. And anyone who actually implemented can can basically pass his implementation against the reference implementation or just transpire it into his language. So I think that lowers the lowest amount of work needed to be done at five different client implementers. And the work of, you know, putting it down into code is only done once in the in the translation from English to code is only done once by the author or someone close it. So I think it's good to move in that direction. But there's spectrum. Yeah, they do get implemented the translation. They get implemented they get translated however whoever's writing it might find it easier to read the English might find it right. The way it is now, whoever implements it first does one implement one interpretation and things yeah that's obvious that must be someone else does another interpretation implements it slightly difference because, you know, yeah. So I think it's nicer. Which, which is heavily I'm just saying if I have any IP, I've completely implemented it and go or C++ and a client. And then it's like, well that's all very well and good but it's not an acceptable IP until you translate it to Python, and it's like but it's expressed here and go and the go works. It's translated to something else that I can't actually run and test. No. But I guess what Martin is saying is, if I understand correctly is that's not the case for like the media and the IP the media and the IP is like, basically under specified in the current format. Yes. Yeah. So, this spec kind of forces you to at least fully specify it and or realize that you can't and that you need to do some more work basically. Yeah. Yeah, I'm just saying that happens in the process by the time. If something gets accepted it will have been implemented and more than one client. We will have found these cases the authors generally aren't in a position to do all of that implementing and testing anyway. So the definition gets more at the tail end when it's actually working how best to express that and currently slowly the IPs make it into the yellow paper, and the yellow paper is is the, the canonical spec. We can change that but trying to get the authors to write canonically up front it's going to be hard. You're going to need an expert to work with the author to do that. And I think that's just going to be even harder to find that person. But if such a person stands up and volunteers. Yes. It's already five minutes over time so we can continue this on each magician so I shared the link it's, it's in the agenda. And I pasted all your comments from the agenda, indeed. Anyone have anything else they wanted to share before we head out. Yeah, to echo what like client said, you can run the full Python implementation for the consensus layer side. Very importantly, the tests that you write for that implementation for that Python spec actually become the consensus tests. And so when we have people build new features, they also write tests and those actually become the reference test. Because I think when we have many different clients and learning EIPs. We don't always capture all of the edge cases and reference tests, even though even when we're kind of like processing our implementations. So it's just another, another component of this process that can be used. Right. Thanks for how I conduct anything else before we wrap up. Okay, I think, yeah, worth noting, I think Europeans, your time will shift before the next awkward devs. We are not shifting the awkward devs time so it'll be at a different local time for you. If you live somewhere where daylight savings time is a thing. Yeah, thanks everyone for coming on. Thanks everybody who didn't drop off halfway through after the merge stuff. And I'll see you all in two weeks. Thank you. Thanks. Bye everyone. Bye. Bye.