ef berlin office:Zk, avms L. 2 s. And quite a lot of other tracks, I think over the next week or 2 weeks, actually, sometime this week before Acd, we should have a summary of everything that happened last week. But for now I think we'd go into a bit more detail on 2 of the tracks with Fusaka being the 1st one.
Transcript
ef berlin office:We had all the specs and open Prs from last week merged into Belinterop devnet 2
ef berlin office:this is a devnet that's gonna be up until next Monday or Tuesday, and the reason we use the bull interrupt name is such that so that we could make faster decisions without needing to go through the Acd process while we were in person. However, those changes will now go through the Acd process and will eventually end up on the Fussaka Devnet 2. Once they've been
ef berlin office:cleared by Acd. That being said, Fusaka Devnet, one is still live and will go down over the next day or 2 unless anyone specifically needs it. So if you do need it, then please speak up, if not, please, move all of your testing to Berlinterop. Devnet 2 and Fusaka Devnet 2 should come up sometime around next week.
ef berlin office:Any questions regarding Fusaka in itself.
ef berlin office:Okay.
ef berlin office:The second track that we spent a lot of time on was gas limit testing.
ef berlin office:I think most of the time was spent on benchmarking clients as well as optimizing performance. I think the outcome was great. We should have a deeper write up. Come out later today.
ef berlin office:However, one of the things we soft agreed to on Friday was to raise the glass limit to 45 million. However, client teams would need to push in the fixes that they have in the performance branch as well as make releases. For this is this something that we can have clients agree to do over the next week?
ef berlin office:Could we maybe get someone from each client team to speak up on this
ef berlin office:in the same room as Harry. So the no. Yeah, yes.
ef berlin office:maybe it's better this way. Yeah. So forget we need at least one fix to get into master until we
ef berlin office:feel like
ef berlin office:We don't feel feel super vulnerable.
ef berlin office:There's a bunch of things where we think like going above 45 is not possible right now without changes to
ef berlin office:without repricings. But I think that, like the biggest point, is like the modex stuff that we are already addressing in in Fusaka.
ef berlin office:So like, from my personal point of view, 45 would be would be okay. But I'm I'm not speaking for the team.
Ben Adams:Now that mine's okay. With 45
Ben Adams:we would like to put out a new release. But I think we're comfortable with the the current release
Ben Adams:and signaling.
Ben Adams:I assume it won't be immediate, anyway.
Ben Adams:and we have a release similar.
ef berlin office:Thank you. Could we have maybe Roman? And then Milan from Aragon.
Roman:Yup or comfortable doing the 45 visits.
milen | Erigon:Yeah. Aragon is ready for 45 as well.
ef berlin office:Awesome. Thank you. Then. I guess we'll share some more info on the benchmarks. We did as well as what bottlenecks we found over the next week, and once the client teams are have all changed their defaults. We
ef berlin office:should the network signal to a higher gas limit.
ef berlin office:Awesome.
ef berlin office:Marcin. I know you've been working a bit more on benchmarks as well as yeah. Exploring gas limits a bit more. Do you wanna talk about it?
Marcin Sobczak:Yes, sure.
Marcin Sobczak:So like, we know like for for the clients, the the worst case is the modex, and it will be repriced in Fussaka. So before Fussaka we have, like
Marcin Sobczak:the we know what is our bottleneck and
Marcin Sobczak:if when modex will be addressed, then probably the next bottleneck is is yet
Marcin Sobczak:so it's it's performing good enough to go for 60 million. But if we aim to, if you want to aim in 100 million between Posaka and the next Harfolk.
Marcin Sobczak:and then we probably need to reprice in in Fussaka, too. And right now I'm crafting more test cases for easy pairing and easy mall to check if it is performing good enough to to keep the pricing as it is, or we need to
Marcin Sobczak:to replace it, too. So far, I I didn't find the test case, which is performing like badly. So
Marcin Sobczak:so so it doesn't look like there will be a need to to touch it in Fussaka.
Marcin Sobczak:Oh, and yes. Do you want Barry to talk about some specific topic about benchmarking or.
ef berlin office:No, I think that I think that's a good update for now. So is, I think, for easy add, we should also have an eip up for that later this week. There is a proposal for what we want to reprice it to.
ef berlin office:And yeah, today and tomorrow, I guess Marcin and a few others would be figuring out. If we do actually need to reprice Ec. Mole or Ec. Pairing going forward. This should help us get to 60 million and further and prepare for post architis. The other topic that came up was log
ef berlin office:logs in themselves, either the pricing or changing the Dev. P. 2, p. Limit. I'm not sure if anyone has thoughts on that, or has time to look into it. I know that there's a proposal from Julio.
ef berlin office:but I don't know if there's any more work being put into it.
Ben Adams:I mean, this sort of kicks in more around 85 million. So we'd still be fine at 60. There's something we potentially wanna
Ben Adams:well, probably definitely want to fix in some way for Usaka. But I I don't think we've come to
Ben Adams:a conclusion on what the right way is. Yeah.
ef berlin office:Is there some specific data point or something we should look at that would help us get to that conclusion, or is it just? We need to discuss this on Acd and come to a solution. There.
Ben Adams:Well, I I think we we need to discuss on Acd. Probably a helpful data point would be to
Ben Adams:actually emit an entire block of logs and then see if there are any performance implications, and then maybe that would help us decide whether we want to
Ben Adams:reprice or increase the peer to peer.
ef berlin office:Okay, got it. I think Pk has a reproducible test for this already. He was able to break snap sync locally, and I'm not sure if he looked at it from a performance perspective. I think he purely looked at it from does this break snap sync perspective? But we should have a test for this that we can reuse.
Ben Adams:Yeah, I I mean, I think there's 2. At the moment there's 2 obvious ways. We can go, and having some benchmarks, would
Ben Adams:probably inform which direction would be better.
ef berlin office:Okay, we can clean that up and get it up to people.
ef berlin office:I think the other topic was Estor with large tree depths. I think this is something we noticed on blocks with Zen contract transactions. It, again doesn't seem to be a major issue, at least for 45 million. However, as we move to 60 and forward, we'll
ef berlin office:definitely be an issue.
ef berlin office:I know. Last week Joachim was working on this a bit. Do you maybe want to go more into what the current status is. Do we have a reproducible test? Were clients able to trace what was going on on? Their clients.
jochem-brouwer:Yes. So this is about the send contracts right?
ef berlin office:Exactly. Yeah.
jochem-brouwer:Yes, yes, So
jochem-brouwer:what it looks like is that it's a. It creates a lot of small contracts. These contracts which are being created are actually proxy contracts, so very small ones
jochem-brouwer:which delegates call into something else.
jochem-brouwer:And what happens is that during this transaction it's a transaction with a gas limit of 28 million.
jochem-brouwer:and it creates 167 small contracts, and it also does a lot of storage. It stores a lot of values. There.
jochem-brouwer:I'm looking a little bit in. If there is something else, something else. Very fancy going on there.
jochem-brouwer:It doesn't really look like it. It really looks like it's a lot of create 2 or create fact, create small great contracts.
jochem-brouwer:and that there's a lot of storage being dumped in there.
jochem-brouwer:and my hunch will be that the caching mechanism of clients that yeah, these, these key values, these are these are like a code.
jochem-brouwer:because you can't really predict before, and that that yeah, in what
jochem-brouwer:what addresses and what key values, we are going to read or write.
jochem-brouwer:And I think that might be one of the problems where we have seen like very slow blocks.
jochem-brouwer:There is a status, or I'm not sure if it was a status or a blockchain test. I think, a state test
jochem-brouwer:which does exactly that. It creates a lot of small contacts.
jochem-brouwer:and it dumps random storage of not random, but it creates a random storage fee there.
jochem-brouwer:and I think what we have seen in clients is that the execution time of this
jochem-brouwer:state test is actually not. Well, it's it's not very alarming.
jochem-brouwer:but on the definite no, no, not the definite. On Mainnet. We saw that it was very slow for some clients.
jochem-brouwer:and this might be because it's like a very huge well, of course, the state database is very big, much bigger than if you would just run a state test
jochem-brouwer:like an empty, or an almost empty database.
jochem-brouwer:and therefore likely on Mainnet, because the storage slots not the storage slots, but the database reads they are. They are cold.
jochem-brouwer:This, then likely takes much longer, and this might be the bottleneck for mainnet.
jochem-brouwer:So what we should likely do is we should run this state test on not a small state.
jochem-brouwer:but actually on a very big state, and then see if we can reproduce
jochem-brouwer:the well. Well, how slow this is on on Mainnet.
jochem-brouwer:and what I'm doing in the meantime is figuring out if there is something else going on. For instance, like we have some locks, maybe there's some extra lock data which might also be a bottleneck.
jochem-brouwer:But in the end I want to create, like a status that we can either completely rerun this entire transaction
jochem-brouwer:isolated or to figure out, okay, if there's something else, something very weird going on, something very exotic in the Evm that I create an even more compact version of this test that you can really squeeze out well, the worst case behavior of the of the Evm.
ef berlin office:That's great. Thank you for that. But okay, that sounds like we are still in the reproducing the issue phase, and not necessarily in a proposal of solution phase, right?
ef berlin office:Yeah, Ben, do you wanna go.
Ben Adams:Yeah. So I I lost Internet at home while I was at the other conference.
Ben Adams:And so when I came back, I I restarted it, and I did run this block.
Ben Adams:Unfortunately, I didn't trace it. But
Ben Adams:I I process 23 blocks.
Ben Adams:which included the problematic one in 600 ms. So I'm not sure exactly what the problem is.
Ben Adams:I mean. Obviously, we we did see a lot of valid. They just struggle with it, but
Ben Adams:might need to look into it a little bit more
Ben Adams:to, you know fully fully drill down on what's happening.
ef berlin office:So I can. I can talk to what the issue was on. Gas.
ef berlin office:Basically, we have a we have a buffer that. We flash to the disk whenever it fills up. After 256 MB
ef berlin office:and this was done synchronously with the the block processing.
ef berlin office:and so often on these blocks
ef berlin office:the buffer would just overflow, because we put a bunch of like the the likelihood to to overflow this buffer to go over the 256 MB is very high.
ef berlin office:And so what what happened? Geth is, we would just spend 2 and a half seconds, flushing this
ef berlin office:flashing this block to the disk
ef berlin office:and our fix, for it is to only flush the basically use a double buffering mechanism. Only flush the inactive buffer to the disk. And
ef berlin office:yeah, and and and switch to the to the to the active buffer.
ef berlin office:Yeah, so that should also take care of this.
ef berlin office:Regardless of what we're doing, like, of how clients actually behave on this block, we should
ef berlin office:be making sure that yeah, this is that we that we have good benchmarks for
ef berlin office:these scenarios where we
ef berlin office:touch a bunch of stuff in the try all over the try and need to recalculate the state root
ef berlin office:and cannot necessarily parallelize it, because it's all in the same storage trap.
ef berlin office:But yeah, we will. We will be working on this and
ef berlin office:and try it out on the perfnet to see if there. Sorry not, maybe also on the perfnet, but mainly on the Shadow fork to see to see how the cans behave.
Ben Adams:So was it bad timing, or would it have always flush the by cash.
ef berlin office:It's it's bad timing, but because it it like normally blocks, don't add that much to the buffer, but these blocks they do so. The
ef berlin office:the chance of them flushing. The buffer is
ef berlin office:much bigger than like a random, normal block.
ef berlin office:but that also means higher the gas limit the worse this gets right unless you make your buffer significantly larger.
ef berlin office:No, we we kind of
ef berlin office:well, when the the way we change it is, we will just flush this buffer
ef berlin office:asynchronously. And so we don't stop the main processing threat on it anymore.
jochem-brouwer:Yes. So this this birth of fish. That's the state tree. Both are right.
ef berlin office:Yes, yes, that is.
jochem-brouwer:Yeah. Okay, cool.
ef berlin office:Okay, I guess we should have some more insights on this by Acd on Thursday. So I guess we continue the conversation there.
jochem-brouwer:Yes, I will make sure that I will check if there are like other weird things happening. But I think also from Mari's point just now that it is indeed
jochem-brouwer:the well, a lot of State rights. And in these random contracts or not, random contracts. But like, yeah, there's like a very big state growth here.
jochem-brouwer:And so, yeah.
ef berlin office:Yeah. Thank you.
ef berlin office:Cool anything else on the gas limit. Topic.
ef berlin office:Cool?
ef berlin office:Okay. If not, then let's move on to peerless testing. I guess we did fusap earlier. That's mainly Prs testing.
ef berlin office:I actually think that was the entire agenda. Do we have any other topics we want to bring up today.
ef berlin office:Yeah. I think we might have an update on history expiry one second on that.
ef berlin office:Yeah, here's Matt with latest on history expiry. Hello, I just want to share some of the things that we talked about last week at interop, just in case anyone who wasn't there was hoping to get caught up.
ef berlin office:Basically, we've talked about what the path forward for the pre merge history expiry is essentially all the clients have implemented what's necessary to do pre-merge history. There's like various things that each client is implemented, and maybe there's not like perfect
ef berlin office:alignment on every feature that every client has. But ultimately, like every client, can run without the pre merge history in one way or another. So we.
ef berlin office:you know, have a path to having all the clients
ef berlin office:choose to run without that history. So basically, now, what we've said as of last week is that clients are free to begin making it the default mode to run without the prune history.
ef berlin office:They can release this at their convenience. In the next week or 2 we'll have a blog post for on the ethereum foundation blog, just sharing that it's possible to run clients without the pre merge history and sharing how to run on the clients that we typically release information with for the the Fork blog posts.
ef berlin office:So we'll publish that in the next couple of weeks. I think most clients have implemented E. 69. This is
ef berlin office:a kind of nice thing to have with respect to starting to drop the history, because now you're not potentially asking a lot of your peers who just don't have the history. You can know ahead of time whether or not your queries are going to succeed. But this is not a requirement for us, you know, having everyone drop the pre merge history. So that's kind of all that we agreed on for how to sort of close a pre-merge. Then post merge. We talked a little bit about what we wanted to happen there, and we kind of agreed that
ef berlin office:we wouldn't
ef berlin office:modify the way the approach with the error file file slightly. There was a thought that with error one, this would be the pre merge, and then we would use the error files as the post merge, and after talking about it a bit more, we kind of realized that it might just make sense to have separate error files for each layer. So the execution layer will have a file that's storing all of the historical data related to it, and the consensus layer will have a file, an error file with all of its historical data necessary.
ef berlin office:So we're kind of recalling these, the Era E and Rsc files. We have a rough idea of what this will look like on the execution layer. Yasik is kind of the person who's been looking at this for the consistently, and I think he has a rough idea, too. So over the next, like one and a half months, we really want to settle on what that specification is, because that's going to unlock the ability of clients to start pruning more history. Basically, we want to move to this place where
ef berlin office:we have a backstop to access the history for whatever reason, and it's been sort of in. It's sort of up to clients to decide. How do they individually want to handle history? Do they want to do rolling history expiry first, st
ef berlin office:or do they want to just support pruning up to suppose cancun? Because, you know, before Cancun, we were using call data for L twos. And there's kind of a lot of call data in that period of time. So we just want to give it to clients to have the ability to make those decisions. And the thing that's on the critical path is just agreeing on what is this like shared format? We're going to have
ef berlin office:to import and export that data
ef berlin office:so hopefully, that is something that will be settled on in the next month and a half by the end of July. And then clients can really
ef berlin office:start to do whatever they want in terms of history. Expiry
ef berlin office:era itself is not like the only long term solution that we want to have. We do want to have some more decentralized and robust way of accessing the history.
ef berlin office:And this is kind of like what the portal network has
ef berlin office:been designed to provide us for a long time, and talking with clients throughout the week, and trying to understand where their portal implementations are, and what some sort needs to be done to get it into a place where it can be relied on by execution. Layers has sort of made us feel like we need to
ef berlin office:from 1st principles. Think a little bit about
ef berlin office:what exactly is Portal trying to provide, because Portal has a long list of features that's very useful for individual like independent portal clients. But there's a very small subset that's useful for execution clients, and some things like serving range queries was not, even, you know, supported by like out of the box given just like how the distribution of blocks are put on the network.
ef berlin office:So we want to make some changes to exactly these, like very this, like small subset of portal features that are extremely useful for the execution clients.
ef berlin office:and then integrate this more tightly into the execution layer. So we want to try and modify portal so that it doesn't rely on an external like client system, and it doesn't share the header proofs for every block. Just to focus on giving value to the execution layer. And then, at a later point, we'll start to like slowly add back these features in as they become necessary.
ef berlin office:So we're having planning to have a public community call about Portal either later this week or early next week to discuss these types of changes. But this is kind of like the types of conversations that we had throughout the week last week.
ef berlin office:And yeah, I think that's pretty much where the history expiry project is right now.
ef berlin office:Thank you. Does anyone have any questions for Matt on the history expiry topic?
ef berlin office:Okay? Then I guess we'd wait for the community call, and in case you had any
ef berlin office:questions until then, please reach out to Matt. If not, bring them up at the community, call
ef berlin office:that being said, I think that's all the topics we had for today. So if there's nothing else people wanted to talk about, then we can end the call a bit early this week.
ef berlin office:Thank you. Everyone.
ef berlin office:Oh, Yup, yeah, okay, that's a buy perfect.
Ansgar Dietrichs:See you bye.
Chat Logs
00:03:24
Ansgar Dietrichs:did pari just confirm we can hear marius, even though the two of them are in the same room?
00:03:35
Ansgar Dietrichs:Replying to "did pari just confir..."
(we can indeed hear marius)
00:09:20
jochem-brouwer:LMK if we need any specific benchmarks, can craft them as state test :)
00:13:30
Ansgar Dietrichs:dumb question, could a more targeted approach not also work, where we just artificially throttle the state access, even when testing on small state?
00:13:51
Ansgar Dietrichs:Replying to "dumb question, could..."
as in, benchmark worst case access performance on mainnet, and then just use that for testing
00:14:03
jochem-brouwer:Antwoord verzenden naar "dumb question, cou..."
this is actually a great idea :)
00:14:29
jochem-brouwer:Antwoord verzenden naar "dumb question, cou..."
we could likely check the trie depth on the state test vs mainnet tree depth and multiply the read time with that ratio
00:15:30
Ansgar Dietrichs:Replying to "dumb question, could..."
right. would be worth empirically validating that that ratio roughly matches real world slowdown on mainnet, otherwise that assumption could skew the results
00:16:32
jochem-brouwer:Antwoord verzenden naar "dumb question, cou..."
yes good point, this assumption might indeed be too simple and cutting corners 😅