Toni Wahrstätter:Welcome, everyone. Today is the November 19th, 2025, the 7th EAP7928 breaker call. We have quite a packed agenda today, I see, because some of us are in… at DevConnect, as you see on the… on the video.
Transcript
Toni Wahrstätter:And we've been discussing for the past hour everything related to Book Club Access List?
Toni Wahrstätter:And we put all those things on the agenda, so…
Toni Wahrstätter:We agree to have a look at the agenda?
Toni Wahrstätter:I would say we can directly start with the first topic, which is…
Toni Wahrstätter:The first topic is actually just a summary of what was decided last year… last week. So last week, we discussed if we should add the gas used values into the block access list.
Toni Wahrstätter:And we decided against it, basically because, things stay way simpler if we just assume clients, order transactions by the transaction index in the block.
Toni Wahrstätter:But I just wanted to put it on the agenda so that we flag this again, and yeah. Does anyone have, different opinions this week that… that you want to share?
Toni Wahrstätter:If that's not the case, I think we can just assume we will not put the gas used values into the block access list and keep it as is.
Toni Wahrstätter:Perfect.
Toni Wahrstätter:Then let's keep it as is.
Toni Wahrstätter:The second agenda item is syncing and pulse.
Toni Wahrstätter:So I would be interesting to hear from client teams what their, strategies are when it comes to syncing, especially
Toni Wahrstätter:which block level access list might be able… will we be able to just drop, and how many should we keep around, at least? And I'm curious what client teams think about this.
Jared Wasinger:I can say for Geth that we haven't discussed this very much internally, outside of the fact that we do see, bowels being useful for removing the healing phase of the SnapSync.
Jared Wasinger:I guess my thinking right now is that…
Jared Wasinger:Probably we will just keep as many bells as we have blocks for, but that…
Jared Wasinger:it's probably a good idea to… and also that I… I do think it should be optional to serve the BALs for historical blocks, because, like, it's optional to,
Jared Wasinger:to serve, like, the SNAP protocol, And then,
Jared Wasinger:Yeah, I guess, my thing is that we just have to decide what
Jared Wasinger:The window… what, like, a suggested window would be to maintain them?
Jared Wasinger:Yep.
Karim T.:Why it will not be necessary to…
Karim T.:To serve a certain block, a certain amount of block,
Karim T.:And this amount will be mandatory. Just like that, we'll be sure that, for SnapSync.
Karim T.:We'll have the reson block, so it should help every time.
Jared Wasinger:Yeah, I mean, it could be mandatory, right? But if you think about, like, the SNAP protocol as it currently is, it's not…
Jared Wasinger:at least to my understanding, it's not mandatory that a node serves the SNAP protocol if it's…
Jared Wasinger:Like, that's… Nuh.
Jared Wasinger:It's also optional.
Karim T.:I mean, if you sell Snap, you should also sell block access, at least I was thinking maybe about something like that, but…
Jared Wasinger:Yeah, I think… I think the SNAP protocol would become obsolete. Like, this would be the… serving the BAL would be kind of like a replacement for SNAP, but…
Karim T.:I'm not sure, because we have different steps in SNAP, so we have the first step, where we are downloading the leaf, and we are reconstructing the tree, so I think this step is still mandatory. And after we have the hill, when we are trying to heal the tree, and maybe the FlatDB, if you have FlatDB.
Jared Wasinger:Yeah, yeah, yeah.
Karim T.:There's a halibut, yeah.
Jared Wasinger:Yeah, yeah, you're right, I… yeah, I… I… I agree.
Karim T.:I, I also had.
Jared Wasinger:I think I agree with you that it would be mandatory to serve… to serve the SNAP and,
Jared Wasinger:the PALs, yeah.
Karim T.:I had also a discussion with Guillaume, he said that
Karim T.:Maybe if we have a reorg during the snap sync,
Karim T.:We will still maybe need a hill, because, block access list is just,
Karim T.:It's not bidirectional, we don't have the…
Karim T.:previous data, we only have the post data, so we cannot roll back a block access list. So if you have a reorg, maybe you'll have to do a heal, but in the happy pass, normally.
Karim T.:Block accesses will be in us.
Toni Wahrstätter:Yeah, Dragon?
draganrakita:I talked with Red Team about persisting balls, and the feedback that I got is the best case for that would be to not persist it at all, and not even serve it on over a P2P.
draganrakita:That would be the simplest solution to us.
draganrakita:If ball is kinda needed for the snap sync, maybe adding, I think Jared, proposed this to have
draganrakita:SnapSync endpoint that will basically fetch ballpark the blockage.
draganrakita:Would probably work.
Karim T.:But…
Karim T.:Block access list, in the block header, we have the block access list hash, so all we can verify the hash if we don't have the block access list, which… the only possibility will be to replace the block and generate it.
draganrakita:In general, you receive the ball from the engine. I would assume that's the point where would we check if this hash is correct or not.
draganrakita:But if we dropped balls from the history, I…
draganrakita:They think that cash is… is going to be there, unchecked.
draganrakita:I think that could be fine.
draganrakita:Because you're basically… Syncing from the top tip to the bottom, and verify all the hashes there.
Toni Wahrstätter:Okay.
Toni Wahrstätter:So just to summarize, so, Jeff and…
Toni Wahrstätter:Yeah, so get… Jared was saying you were…
Toni Wahrstätter:Planning to keep the bars around, and you make it optional for serving for historical blocks.
Toni Wahrstätter:And Dragon was just saying he would prefer to drop them and not serve them at all, or at least keep a few of them around, am I correct?
Toni Wahrstätter:With a few, I mean, enough for SnapSync.
draganrakita:Yeah, I think the team said that they're fine with dropping them all.
draganrakita:But I'm not that familiar with that part of the code, so I'm just, like, transmitting the message that I received.
Toni Wahrstätter:Okay.
Toni Wahrstätter:Yeah, this sounds like we would want to keep at least enough block lab access lists around for, like, the week's objectivity period, so, like.
Toni Wahrstätter:Am I… am I correct here? Like, 2 weeks of block lab access lists? Is that what is, like, needed in the worst case?
Jared Wasinger:Can you… can you expand on… On the rationale for that?
Toni Wahrstätter:Like, what is the… what is the, point in the past when you would start Snapsing?
Jared Wasinger:Oh, yeah, I don't know, I mean, I guess it kind of depends on what the…
Jared Wasinger:After you download the entire, initial state.
Jared Wasinger:Like, how long… the amount of time that you would keep the balls around for would depend on, like.
Jared Wasinger:How long it would take to download the initial, like, state snapshot.
Jared Wasinger:And how far you would be behind, so it's kind of, yeah, like, I mean, it's dependent on…
Jared Wasinger:the… However fast the given node can sync.
Karim T.:I think, in general, it's, less than a day.
Karim T.:That's true.
Jared Wasinger:Yeah, 2 weeks would be a long time.
Karim T.:Hmm.
Toni Wahrstätter:Okay, perfect.
Toni Wahrstätter:Any other client has an… with an opinion on that?
Ben Adams:Isn't that normally, sort of, 64 blocks, or around there? Because otherwise you stop getting responses from,
Ben Adams:You need to be quite close to the head to discuss it.
Ben Adams:So, so a two-week period would be fair enough.
Toni Wahrstätter:Okay, this is, this is good, because this is, then way less, data than I initially expected.
Toni Wahrstätter:Yeah, I think it's good that we, have discussed it already. Maybe we can also…
Toni Wahrstätter:Bring it up in two weeks for the next breaker call, just to, confirm that again.
Toni Wahrstätter:And yeah, use the time in between to look into this, more carefully.
Toni Wahrstätter:Perfect. Then let's just move on with the next agenda item.
Toni Wahrstätter:Which is block lab access list metrics and Tracer. So basically, it would be great if we had the metrics that Katya proposed a few weeks ago, if we can get them
Toni Wahrstätter:as early into the protocol as possible, so best case that we have them in definite zero already.
Toni Wahrstätter:careers will…
Toni Wahrstätter:clients think about this, so how far are we on those metrics? Have you already looked into this?
Toni Wahrstätter:They assume… This means no.
Toni Wahrstätter:Yeah, I'm not… I'm not sure, in terms of priorities, what we should do here, if we should…
Toni Wahrstätter:get the metrics already in Definite Zero, or if we want to focus on having, clients running smoothly on Definite Zero without the metrics set, then we're fine with that.
Toni Wahrstätter:And on a related note, I think Felipe has a proposal for including the block accesses in the logs, just as Raph does right now.
Toni Wahrstätter:Felipe, do you want to expand on that?
devconnect:Yeah, I can… I can expand a little bit.
devconnect:Right now, debugging is… is quite tough, and so I think that if in debug logs we, for…
devconnect:Block-level access list mismatches. If we can get the, the bells, logged, it would help significantly on the testing side.
devconnect:And so this is… this is more just of an ask to… to try to get those as debug logs, because right now, for a lot of clients, we just have the…
devconnect:The hash mismatch when it's invalid?
devconnect:Yeah, and that makes things significantly difficult.
devconnect:to, to find the issues, so…
Toni Wahrstätter:Yeah, just as Penny says here in the chat, I'm relaying that.
Toni Wahrstätter:Putting the whole diff into the log, and not just the hash mismatch, that would be great.
Toni Wahrstätter:Perfect, and just to summarize those two points, I guess we are… Sorry, Mario?
devconnect:So, when you, when you say, diffs only, it's… you still full, print the full block-level access list only when diffs happen? Or do you mean that we should only print the diff?
devconnect:Of the block-level access list against the expectation, or could you expand on that?
Łukasz Rozmej:You only print the diff, so print the things that are missing in the access list found, and print the things that are excessive in the access list constructed, right? So don't print everything.
Łukasz Rozmej:Because access lists can be huge, and diffs can be, like, one or two addresses.
devconnect:Sure, sure. I think then the log should be very clear, like, which one has the diff, right? Whether it's the test vector or the client.
devconnect:Because sometimes that can't… that's not clear, so as long as we agree on that, then… yeah, I think the diffs make the most sense, for sure.
Toni Wahrstätter:Perfect, thank you very much.
Toni Wahrstätter:Then, let me quickly summarize that. So, we will likely not have the metrics and the definite zero.
Toni Wahrstätter:But we will definitely try to put the diffs into the logs, as soon as possible, just to make testing and stuff easier.
Toni Wahrstätter:Great.
Toni Wahrstätter:There… okay, what… sorry, let me quickly read the chat.
Toni Wahrstätter:What… what do we do with the orders? Yeah, this is a good point.
Toni Wahrstätter:Do you have an idea, Lukash?
Łukasz Rozmej:I would have to think about it. Is the order important?
Toni Wahrstätter:Yeah, it is important because we, of course, enforce some ordering.
Toni Wahrstätter:But, yeah, I'm thinking, is it… is it that important to have that in the logs?
Toni Wahrstätter:Because we… if we have the diff anyway.
Toni Wahrstätter:the problem might not be in the ordering, right? So I assume there will not be
Toni Wahrstätter:Back in the order.
Ben Adams:And overly?
Ben Adams:Generate a different option.
Ben Adams:We just have a generic…
Jared Wasinger:If the ordering's different, can't we just have a generic error message in the logs, and it, like…
Jared Wasinger:You don't really need the… the BAL diff won't…
Jared Wasinger:Even be necessary if that is the source of the error.
Toni Wahrstätter:Right, yeah.
Toni Wahrstätter:Yeah, that's my feeling, too. That if the ordering is wrong, we might be fine with just having a special message there.
Toni Wahrstätter:And for all other issues with the bar, we will include the… In the law.
raxhvl:The ordering itself could be part of it, let's say.
raxhvl:We expect a certain order, and this is the given order. It can be formatted like a bit.
Toni Wahrstätter:Perfect. Is there anything else we should, discuss regarding
Toni Wahrstätter:making life for testers easier? Is there anything else?
Toni Wahrstätter:Mario or Philippe that you could think of that we might need? Stefan was just saying, any updates on including generated block access lists?
Toni Wahrstätter:In the debug that block call, Does anyone have insights there?
Jared Wasinger:I made some progress for implementing it in Geth,
Jared Wasinger:And I mean, I can definitely see why it's useful.
Jared Wasinger:But it also… it also kind of makes the code path a bit convoluted, because now, when we are executing.
Jared Wasinger:And we receive some error from a transaction, we can't just immediately stop executing the block and terminate, we have to, like…
Jared Wasinger:execute every single transaction and reconstruct the block access list, and it just creates, it makes an already complex code path just, like, a bit more…
Jared Wasinger:difficult to reason about, so I didn't finish implementing it yet. I mean, it can definitely be done, but it's…
Jared Wasinger:Like, we haven't… on the get side, we haven't…
Jared Wasinger:There hasn't been that much, like, review on the code yet, so… it's, like, adding another piece of complexity on an already…
Jared Wasinger:Significant change to the,
Jared Wasinger:To the execution and, state route calculation path, but… no.
Jared Wasinger:But that being said, I mean, I only spent, like, a day on it, so there's a… there's…
Jared Wasinger:Maybe I'm… Overstating how… Invasive it would be, but…
Jared Wasinger:That was just… this is just my impression from having done a bit of, like, cursory… investigation into it.
Toni Wahrstätter:Has someone else from other client teams already worked on that?
Karim T.:On the zoo side, we have it, so… should be, should be okay.
Toni Wahrstätter:Awesome. Anyone else?
draganrakita:From red side, we're executing all the sections, and recreating the ball, and just comparing it at the end.
draganrakita:So we will not fail if…
draganrakita:No, we'll fail if we fetch something that's not existing in the ball.
draganrakita:But we will not fail if the change is different than expected.
draganrakita:So, there is some cases where we will We maybe could know beforehand.
draganrakita:But we would check it at the end of the block.
Stefan Starflinger:My question is, are you providing an endpoint for the get bad blocks, and including the generated block access list that you generated in this endpoint as well, so that we can use it for debugging?
draganrakita:It's… it would be easy to add, it's just executing and recreating the ball.
draganrakita:But I'm not sure if we have the endpoint for the bad block.
Stefan Starflinger:Would be great if you could, get back to me on that.
Toni Wahrstätter:Yeah, this would be great. Maybe, maybe, would be great if, Stefan, if you could drop a message in the blog access list, Discord.
Toni Wahrstätter:Just that we have it, yeah, somewhere written there.
Toni Wahrstätter:So that we can refer clients to that message, then.
Toni Wahrstätter:someone from MetaMind, who can give an update on… on that topic?
Ben Adams:I don't expect this, request.
Łukasz Rozmej:Oh, sorry, I don't know. Mark is the only one that has the newest knowledge. He's working on it, but I don't see him on the call.
Toni Wahrstätter:Okay, yeah, no worry.
Toni Wahrstätter:Anything else regarding, things we might wanna have for testing?
Toni Wahrstätter:That we should discuss today?
draganrakita:Just to say that the last testing feature is really great. It fixes all the bugs.
draganrakita:So, it's perfect.
Toni Wahrstätter:Yeah, it looks like Ref is quite far when it comes to testing. We can… we can directly go into the client updates, so how are all our clients doing when it comes to testing with the latest release?
Jared Wasinger:Yeah, I've been mostly working on the performance side, so I haven't, been working on the, getting conformance with the tests in the latest release yet, but…
Jared Wasinger:It's… on my list.
Karim T.:So, on Bezos' side, we fixed several issues, thanks to the new test, but there are still some issues in the test suite, so we'll see with the new version.
Karim T.:We are also… we have a first version of the stateful computation background that is working, but we are trying to do some improvement on it.
Karim T.:And we are starting to work on the perfect parallelization using BAL.
Karim T.:It's all…
Toni Wahrstätter:Thank you very much.
draganrakita:From red side, here on the tests, I think one is failing, but 40…
draganrakita:40 faulty ones were fixed.
draganrakita:Yep, that's… mostly it.
Toni Wahrstätter:Someone from Aragon here?
Toni Wahrstätter:I don't think so.
Toni Wahrstätter:Let's continue on with NetherMind, or should we… or do you want to wait until Marchin will… or Mark will provide the update?
Toni Wahrstätter:Or do you, Ben, or Lukash, do you have insights into… okay, it just so pumps up, perfect.
Toni Wahrstätter:Yeah, there is… there is this one topic that has been a recurring theme over the last
Toni Wahrstätter:months, basically, about the REIT locations, or the state locations in Nepal.
Toni Wahrstätter:And… The latest I've heard so far was client teams very much were for including the relocations.
Toni Wahrstätter:Because it enables parallel and I.O.
Toni Wahrstätter:And this is the question, like, how would we best, test it? I know that Lukash has suggested that we just test it in actual client implementation, so, for example, on ketosis or something.
Toni Wahrstätter:Is this… Something that other clients think is… might be the best thing to do, to actually
Toni Wahrstätter:determine if it's worth adding the state locations, because, of course, they add a lot of data to the block. It's, like, 50% of the…
Toni Wahrstätter:Ball size comes from the read locations, the state locations that are not changed.
Toni Wahrstätter:And we have been discussing it in a bunch of breakout calls, but We haven't really gotten to…
Toni Wahrstätter:benchmarks that were, like, super clear when it came to adding the state locations. And especially, we have to think about higher gas limits, like.
Toni Wahrstätter:beyond 100 million, and if it makes no sense, when we are at those high limits, to have the state locations, or if the value of having the state locations in the balls is kind of declining over time with an increasing gas limit.
Łukasz Rozmej:So, that's a very good question, and exactly we can do different things. We can…
Łukasz Rozmej:Blindly at state locations, we can blindly exclude them. We could try to…
Łukasz Rozmej:Implement kind of both and benchmark it on different… on different versions and different… different kind of blocks.
Łukasz Rozmej:We can all… then the last thing, we can always try to be smart, and by smart, I mean…
Łukasz Rozmej:Do something like…
Łukasz Rozmej:include them conditionally, right? So, by conditionally, I mean, we could have some heuristic, and that heuristic is that if the transaction does at least N reads, right, or something.
Łukasz Rozmej:Then you include them, otherwise you don't, right? So, there is way, way…
Łukasz Rozmej:different strategies to target different cases. I think we should be aware about the size increase, so generally, I'm a bit hesitant about including them blindly.
Łukasz Rozmej:But on the other hand, if there is an edge case transaction that does a huge amount of reads.
Łukasz Rozmej:And okay, with 16 million gas limit, it's still…
Łukasz Rozmej:it's still bounded, so it's not that bad as without it. But even then, maybe there's some threshold where it's worth including, so…
Łukasz Rozmej:We can do multiple things, and, depends how…
Łukasz Rozmej:How much precise in-tuggling bottlenecks we want to be.
Toni Wahrstätter:Yeah, my… my concern with, having, like, heuristics for the reads is the following.
Toni Wahrstätter:it's like, imagine disk I.O. becomes a bottleneck, then the worst-case block would not only be worst case in terms of having a lot of S loads, but also having additional data in the block. So it's basically, like, for blocks that are already, like.
Toni Wahrstätter:very bad to execute. We would be like, okay, let's add additional data to exactly those blocks, and the blocks that
Toni Wahrstätter:Our lightweight will probably not have the read values in them.
Toni Wahrstätter:You know, it's like…
Toni Wahrstätter:It feels almost, like, counterproductive to be like, okay, our heaviest block will now also include additional data.
Łukasz Rozmej:By heaviest, by network transmission size? That's what you mean, or…
Toni Wahrstätter:No, by just workload for I.O, for example, assuming, for example, a block that does only S loads.
Łukasz Rozmej:Okay, but that block now is, can… those S-loads can be parallelized, right? Like, completely.
Łukasz Rozmej:At least for…
Toni Wahrstätter:Yeah.
Łukasz Rozmej:worse transactions, so actually, we kind of alleviate the I.O,
Łukasz Rozmej:bottleneck here, by including the reads for those.
Łukasz Rozmej:So, my expectation would be the opposite, that we actually have the worst blocks, that
Łukasz Rozmej:Let's say we have, I don't know what gas limit are we targeting? 300?
Łukasz Rozmej:Okay, 300 gas limit, so 300 divided by 16, that's around 19 transactions, worst, and if they are all doing loads, so… we could…
Łukasz Rozmej:you know, parallelize that. The networking is the bad thing, right? That we get hit by this huge increase in balls.
Łukasz Rozmej:So…
Łukasz Rozmej:I don't know, it's hard to say without the numbers, it's hard to say without benchmarking, it's hard to say without…
Łukasz Rozmej:Additional complexity of clients that will hit us here and there.
Łukasz Rozmej:But… Yeah, I think, like, in most cases, we don't need readables?
Ben Adams:So they would be just an overview.
Łukasz Rozmej:overhead for now.
Ben Adams:working in the proposal.
Łukasz Rozmej:But here and there might be a transaction that we will say, oh, now this transaction, all transactions are finished in few milliseconds, and this one transaction now tails for next 100 milliseconds. And if we had balls for it, read balls for it, it would just…
Łukasz Rozmej:Be 10 times less.
Łukasz Rozmej:So that's my guess.
Łukasz Rozmej:But, without measuring it, it's hard to say.
Ben Adams:One, kind of objection I have to the parent.
Ben Adams:Read slots is they don't… sorry, the read locations is they don't have what transaction they're ready.
Ben Adams:So they're all just, lexographically ordered, which…
Ben Adams:is the equivalent of, like, a random lift, so you…
Ben Adams:you can't overlap the running of transactions with the I.O. You can't… you can't say, alright, I've loaded the
Ben Adams:data for the first 10 transactions, so I can start running them now, while I'm still loading the data for the following, you know, 100 transactions.
Ben Adams:You don't know.
Ben Adams:what date you're reading? Obviously, there's a size buffoon going back into the.
Toni Wahrstätter:Yeah, that's true.
Toni Wahrstätter:I mean, this is in general something we can discuss, also alongside having the state locations in the house, if it would make sense to add transaction indices. Of course, the problem here is, as soon as we add transaction indices, we cannot aggregate them as well as we can today, because today, if we have a right
Toni Wahrstätter:to any storage slot, no matter in which transaction index, we don't include that right as a read anymore.
Toni Wahrstätter:Of course, if we would now add transaction indices to REITs.
Toni Wahrstätter:Which are not needed for parallel execution.
Toni Wahrstätter:then we would have to kind of deduplicate those entries, and you would have the same storage slot as a write and a read, with potentially different transaction indices. So, my expectation would be that the block lab access list would definitely grow in size by…
Ben Adams:If… if we assume, that you parallelize transactions in order, so you… the first transaction you run is…
Ben Adams:The first transaction, and the second transac- you know, the second transaction, even if it's happening at the same time.
Ben Adams:It's the second one. Then you could…
Ben Adams:achieve the same effect by just marking the first transaction that that slot was read in. Not, not every transaction that's read. So you just need to say, oh, right, this is read in transaction 1, this is the first time it's read. Transaction 1, transaction 50, or whatever. And then you can
Ben Adams:You could shed… you'd have enough to start scheduling. You wouldn't need to record, you know, this is read by transaction 1, 5, 10, 11.
Łukasz Rozmej:Yeah, so… exactly, so if we had write first and then read, we don't need to include the read one, and if we had read first and write later, then yeah, we would have duplication.
Łukasz Rozmej:But we, again, wouldn't have to…
Łukasz Rozmej:Only once, right? So only one duplication. We don't have to mark reads for subsequent transactions, like, only the first occurrence of a read.
Łukasz Rozmej:In the block.
Toni Wahrstätter:Yeah, this makes sense.
Toni Wahrstätter:Then I guess the question is, is this worth the complexity?
Toni Wahrstätter:I guess this might also depend on the numbers we are looking for, right?
Toni Wahrstätter:So, with numbers, I mean the benchmarks that we might still need to get, yeah.
Toni Wahrstätter:Because, of course, this would be, like, a worst case that, imagine, we have,
Toni Wahrstätter:A transaction that is at the very end of the block, and also scheduled to be executed at the very end of the block.
Toni Wahrstätter:But… The prefetcher starts prefetching the store slot of that transaction first, because they are lexicographically first.
Toni Wahrstätter:And then we would end up with execution not having the cache ready to be consumed.
Ben Adams:I think it'll become more of,
Ben Adams:Leaving a lot on the table as the… as the blocks get bigger.
Ben Adams:That's the slots get shorter.
Ben Adams:Because whether, you know, where you want to do a balustr.
Ben Adams:Or, lovely better.
Toni Wahrstätter:Yeah, I totally agree, but this is definitely…
Toni Wahrstätter:My only concern would be complexity here.
Toni Wahrstätter:like, will we find… like, this rule that, you and Nucha suggested, that…
Toni Wahrstätter:If there was a write with a lower transaction index, then the read would just not be included at all, like we do today. We don't include it if there is a write.
Toni Wahrstätter:And then, like, if there is a read with a higher transaction index, with a lower transaction index, then we would actually include the read with the index, kind of telling the client this is something that needs to be prefetched right at the beginning.
Toni Wahrstätter:So what do clients think about complexity here?
Ben Adams:I also think, if they're in the same transaction, so you do, do an S load first, and then you do an S store later in that same transaction, that would only need to be the lead, because you don't… sorry, that would only need to be the right.
Ben Adams:Because… The reason implied. So it would only be if it was… I read it.
Ben Adams:in a… Earlier transaction that had an array.
Ben Adams:Hopefully that shouldn't…
Ben Adams:necessarily add too much superposition. I don't know how much overlapping there would be an exact job.
raxhvl:We could also wait for some data to…
Toni Wahrstätter:Yeah, go ahead.
raxhvl:We could also wait for some data to come in. I think, every client could have their own niche parallelization strategy, and we could
raxhvl:see, like, what part of execution is idling on I.O.
raxhvl:And once we have more data.
raxhvl:This would also vary a lot based on investor advice, so we could see, like, where are the rare, limits for education idling.
raxhvl:And then maybe…
raxhvl:For lower draftsmen, maybe under 100 million, we're not even need it now, so we could just add the complexity as we go later.
Toni Wahrstätter:Yeah, I can see that point, but I think… I think, like, we should definitely design a system to work, like.
Toni Wahrstätter:to be, like, future-proof. For example.
Toni Wahrstätter:Ready to support, like, 300, 500 million guests, and so on.
Toni Wahrstätter:So, this is, like, why it's so difficult right now to tell those things, because we have no experience with such high gas limits, and as Ben says.
Toni Wahrstätter:the REIT indices, so having transaction indices mapped to the REITs, might be becoming more important with increasing gas limits, because
Toni Wahrstätter:Of course, the worst case would then just be a transaction at the very end, and the prefetcher starts prefetching everything that is read in that transaction, although it's only needed at the very end of the block.
raxhvl:I agree. Yeah, we should, we should target,
raxhvl:Like, getting numbers, or at least hearing them alone.
raxhvl:And we could also see if we can, like, optimize our parallelization strategy.
raxhvl:To work around this.
raxhvl:And only the larger output would be to add more complex material.
Toni Wahrstätter:Right, yeah.
Toni Wahrstätter:Yeah, I guess at least from the spec side, if I think about the specs, this shouldn't be too complex, because this would be a change to this
Toni Wahrstätter:quite isolated in the block access list builder. Basically, the part of the code that takes the… everything that was tracked in the state, and then builds the aggregated block access list. So from a spec side, this should be
Toni Wahrstätter:Okay-ish to do, and not, like, super complex.
Toni Wahrstätter:But of course, we were just talking about removing the read values, and now we are like, okay, let's add some more inputs to the reads. I get the argument that, okay, reads make more sense if we
Toni Wahrstätter:Have, like, this additional, 8 bytes, which is, like, the transaction index attached.
Toni Wahrstätter:But yeah, I would say we just, wait for numbers, and then we decide upon those things.
Łukasz Rozmej:One comment about, having some data about big blogs. So Nevermind, Camille from Nevermind, with Carlos are doing those, some experiments where they have a tool that combines
Łukasz Rozmej:new payloads from mainnet, they're using from mainnet, that combines new payloads into very big new payloads. So, they try to aim around 1 giga for new payload, for example, but this is more or less configurable.
Łukasz Rozmej:And then they replay those extremely big blocks on the,
Łukasz Rozmej:on the notes, to get some performance metrics from it. So, this could be adapted to get even more better, maybe real-life numbers, or if you want to check anything there.
Łukasz Rozmej:That sounds like a good idea, to, to get some experience with.
Łukasz Rozmej:Real-like, blocks that are around 500 or 1 giga gas.
Toni Wahrstätter:Okay, yeah, this is interesting.
Toni Wahrstätter:So this is… this is our… this is separate from Bloatnet, right? Because this might also be something we want to test, like, how will everything behave on a bigger…
Łukasz Rozmej:So, currently, they were working on mainnet, because they wanted to get, I think, like, more mainnet-like numbers.
Łukasz Rozmej:But this could be used on BloatNet, too, with no problem.
Łukasz Rozmej:But there is a lot of setup, because each client has to have a correct snapshot, etc, so it can be rewinded to that point of time, so it's… it's,
Łukasz Rozmej:It's quite a manual process, or at least the… maybe not manual, but there is a lot of manual, like, preparation in order to automate this process, and it's not really very, like,
Łukasz Rozmej:Easy to just adjust it on the fly, right?
Toni Wahrstätter:Right, yeah. Okay, but this is good to know, thanks.
Łukasz Rozmej:I think they will present it somewhere tomorrow about it, on some Nevermind-related presentation, if you want to.
Łukasz Rozmej:No more.
Toni Wahrstätter:Yeah, maybe you can drop the link, if possible, in the botlab Access List breakouts.
Łukasz Rozmej:Fuck.
Toni Wahrstätter:channel, or… or you send us a DM, yeah.
Łukasz Rozmej:Yeah, I can, I can ask them,
Łukasz Rozmej:For, for the link, too.
Łukasz Rozmej:Okay, thank… no worries. Dragon?
draganrakita:Yeah, I'm just saying, commenting on my gut feeling.
draganrakita:the block journal and execution journal is not just loading the data, it is some CPU. So, having what needs to be read from the database makes sense, because we want, when that CPU happens on…
draganrakita:4, 5, 8 threads, whatever we have. We want to have something done in the background.
draganrakita:So, having what needs to be re-read
draganrakita:It seems that it's going to be impactful.
draganrakita:Knowing order, but needs to be read
draganrakita:I don't think it's going to be that important.
draganrakita:Because even if some thread needs to wait for something to be read.
draganrakita:We still… we are not going to be… Bottlenecked by that much.
draganrakita:It's not something that's going to take, 100 millisecond.
draganrakita:So, there will be slightly… there will be a need for mechanisms that will prioritize.
draganrakita:The… the reads that are happening now, in comparison that older it.
draganrakita:But I'm not sure if the benefits, like, this is just a gut feeling, like my comment. I'm not sure if adding the indices to the reads are going to matter a lot.
draganrakita:At least that's… this is how I'm thinking now. Without data, this is just, like, hypothesis.
Toni Wahrstätter:Yeah, this makes a lot of sense, and this was also, like, where… what I was thinking about, that maybe as long as you can do parallel I.O,
Toni Wahrstätter:There's one process busy anyway, and you are just, like, executing in the meantime, and then you would at least have everything in cash already for the last transactions, and for those first transactions, where it would be handy if you have them ready in the cache.
Toni Wahrstätter:you would just do some prioritized I.O. for exactly that transaction, so you can continue executing.
Toni Wahrstätter:But this, yeah, I mean, we need the numbers for that, we desperately need them, but I guess… It's…
Toni Wahrstätter:Yeah.
Toni Wahrstätter:not so… not so easy to tell, because I also have hard times, telling what is the actual, efficiency gains we get from parallel I.O. versus splitting, distributing transactions over threads, and then having this implicit parallel I.O. to
Toni Wahrstätter:Or I should better frame it as batch I.O. versus parallel I.O, because if batch I.O. is really, like, super powerful.
Toni Wahrstätter:then it might… and, for example, I could… you could imagine, like.
Toni Wahrstätter:I.O. would take a fraction of the time that the actual execution takes.
Toni Wahrstätter:Then, having the indices might not make much sense.
draganrakita:Just additional comment, reading in…
draganrakita:when you know the data is ordered, at least MDMEX have, like, the cursor that can… it's faster to read, like, when you have ordered data.
draganrakita:So, having a batch of all the reads that needs to be read with the sorted order is going to be faster than reading it by randomly.
draganrakita:I'm not sure by how much.
draganrakita:That's, like, big information here, but I would say 1.5 to 2 times faster.
Toni Wahrstätter:Awesome, yeah.
Toni Wahrstätter:So, what do we think about, those numbers that we actually need? So, what would be the next steps?
Toni Wahrstätter:that,
Toni Wahrstätter:what are you guys planning on doing so that we might get those numbers from actual implementations? Would you need anything from…
Toni Wahrstätter:Would you need any specific tests, or…
Toni Wahrstätter:Like you said, with the increased block sizes and that kind of stuff.
Toni Wahrstätter:Like, how would we actually get to those numbers that we would want to have, to have, like, certainty over adding
Toni Wahrstätter:The state locations, or even adding, like, transaction indices to the state locations.
Toni Wahrstätter:Yep, Ben?
Ben Adams:I think in the guests…
Ben Adams:testing, or the… or maybe the BlakeNet channel. So Joachim came up with… he came up with this little…
Ben Adams:Tests that were performing quite badly.
Ben Adams:With S-loads in their stores, and it seems…
Ben Adams:They're exactly the access initiative address, so we may already have seen that.
Ben Adams:But I'll look into our family.
Toni Wahrstätter:Have other clients looked into… Yeah, go ahead.
Karim T.:I just wanted to say, maybe we need multiple devnets with different configuration at the same time, and
Karim T.:And we can see the different performance.
Toni Wahrstätter:Yeah, of course. This comes, of course, with some overhead that would be nice if we can avoid, like, having multiple defnets with different configurations.
Toni Wahrstätter:I totally get the argument that it's, like, difficult to simulate it in a vacuum without having actual client implementations running.
Toni Wahrstätter:Yeah, I'm not sure. What do other buy-in teams think about this? Should we actually…
Toni Wahrstätter:go for different definets, or would it be enough to have, like, a DEFNET that has it implemented, and we measure everything from there?
Karim T.:Maybe not by configuration, but for example, one DevNet with a big state, one DevNet with a… as it may be a shadow fork of mainnet to see the current performance.
Karim T.:just some kind of configuration like that, and in one, for example, in the blood devnet, we can have multiple BISU with different flags, but .
Ben Adams:Yeah, would it be useful to be able to pass in a flag to turn off any block-level access lists,
Ben Adams:Optimizations, like, taking advantage, not doing any preloads, etc.
Ben Adams:And then, not doing any parallelization.
Ben Adams:And then literally run the same chain.
Ben Adams:one with the flag on, saying, don't do any block. I mean, obviously, you still have to, perhaps, verify it.
Ben Adams:And then you'll… you'll see this is the effects from the optimizations, you know, parallelization, doing…
Ben Adams:Loading state, etc, and here's the effect without the wheel.
Ben Adams:We'll see what the appliance does look like.
Toni Wahrstätter:Yeah, I would agree.
Toni Wahrstätter:just a little bit hesitant because of the complexity there. I don't want, like, clients to have, like.
Toni Wahrstätter:overly complex, implementations on the DevNet.
Toni Wahrstätter:Not sure, if this is actually super complex for clients.
Toni Wahrstätter:To have, like, a flag that turns on and off the state locations.
Toni Wahrstätter:Basically, the state locations are always there, but that turns on and off if we actually use the state locations.
Ben Adams:Sort of have that for,
Ben Adams:I think both Netherland and REC, and maybe some others, so we do cache and pre-compile results, and in order
Ben Adams:measure that for, HERFNAP.
Ben Adams:And for the, the benchmarking, we have a flag that literally switches off parts of
Ben Adams:Switches off the optimization, so we can get access to the raw underlying, you know, what is the performance of this reconfile.
Ben Adams:So, I think it's… I think it's possible, if it would produce, worthwhile results.
Toni Wahrstätter:Maybe a direct question to Jared and Dragan and Kareem, what do you guys think about this?
Karim T.:I don't think it will be difficult to have some flags,
Karim T.:We have already some flags, we already started to have different flags with different configuration for block access list, in order to be ready for this kind of test, so I don't think it will be difficult.
Karim T.:To add more flags, or two.
Karim T.:To add more configuration.
draganrakita:I'm not exactly sure.
draganrakita:What flags are we talking about? Flags that would remove prefetching of the reads?
draganrakita:Just that one.
Toni Wahrstätter:Yeah, exactly.
Ben Adams:And, and then the… Paralyzation, analyzing, if we've done it.
draganrakita:Can you repeat that, Ben?
Ben Adams:And any parallelization, or, sort of using the…
Ben Adams:SCOR changes to then use that to parallelize all the transactions.
Ben Adams:I assume.
Ben Adams:We'll hit this up.
draganrakita:Okay, removing parallelization of the sections. Okay.
draganrakita:Those two flags.
Ben Adams:So you, so you, so you can do a,
Ben Adams:This is without block-level access, this is with block-level access.
draganrakita:Makes sense.
Ben Adams:And then, you know, the next day, you put gas to 500.
Toni Wahrstätter:Have you been looking into this, or thinking about this already, Jared? Is there any concerns that you would have regarding adding a flag that turns off prefetching the state location?
Jared Wasinger:Yeah, we can do it,
Jared Wasinger:And it should be possible for the transaction parallelization as well. I'm… I guess I'm just not… fully understanding
Jared Wasinger:how the insights gained from the… so, sorry, I kinda… I… my attention kind of trailed off for a bit, but, I… I'm not really understanding how the… the insights gained from this
Jared Wasinger:Will… Help us determine whether the…
Jared Wasinger:current read format is sufficient, or if we need to add the transaction indices. I mean, yeah, I can… I can add flags to disable these various, the… the… these… these flags are possible, but it just doesn't…
Jared Wasinger:I'm struggling to connect the dots in my head.
Toni Wahrstätter:Yeah, I think there are two different things. One, we want to know the difference between having the state locations versus having… don't having the state locations, and for that, a flag that just, decides if we use the state locations from the ball in the prefetcher or not.
Toni Wahrstätter:Would be enough.
Toni Wahrstätter:regarding the transaction indices, this will be more complex, and I think we should more… first focus more on the first, determining if the state locations are worth it, because
Toni Wahrstätter:if, for example, Jagan's feeling is, right, that
Toni Wahrstätter:We might not need the transaction indices for the state locations.
Toni Wahrstätter:Then we will find that out with only that flag, with only the flag that turns on and off the state locations.
Toni Wahrstätter:If we find out that
Toni Wahrstätter:at a higher gas limit. State locations might be helpful, but only if they actually included the transaction indices.
Toni Wahrstätter:then this is different, but I would assume, we can just continue with having this flag that only targets, basically, what the prefetcher does,
Toni Wahrstätter:Either it runs with the state locations, or you would just run the prefetcher as it runs today.
Jared Wasinger:Yeah, okay. So, for us, it would be easy to… to enable or disable
Jared Wasinger:The asynchronous loading of the read locations, but…
Jared Wasinger:The prefetcher as it runs today is not…
Jared Wasinger:a thing, post BAL, forgetth, but yeah, otherwise… Okay. Yeah.
Toni Wahrstätter:Okay, yeah, then, my fault. I was just assuming that the loading from disk of the state locations would be prefetching, but yeah.
Toni Wahrstätter:Sorry for that confusion.
Jared Wasinger:No, it's okay.
Toni Wahrstätter:Awesome.
Jared Wasinger:Wanted to clarify.
Toni Wahrstätter:Yeah, then let me… let me quickly summarize. So, we decided today that we do the…
Toni Wahrstätter:we talked about, SnapSync, that we might keep some buzz around, at least to enable SnapSync, and then we also serve them for SnapSync.
Toni Wahrstätter:Regarding the logs, we decided that we all put… so all client teams put the divs into the logs, so that, this makes life of testing team easier.
Toni Wahrstätter:And finally, we also decided that we want to do, a defread with potentially adding, or with adding a flag that turns using the state locations on and off in order to give us numbers on how important the state locations actually are.
Toni Wahrstätter:Great. We are already at the end of the call. Is there anything else we should discuss?
Karim T.:I just had a question, are we talking to add this flag for the next DevNet, or another one?
Karim T.:Because, for example, in Bezu, we didn't implement it yet, the prefetching from the block access list, so… just to see if we need to pre-authorize that or not.
Toni Wahrstätter:Yeah, that's a good question. My feeling would be we would probably want to have that on DevNet1, just because we… the earlier we get those numbers, the better. Or for DevNet 0, sorry.
Toni Wahrstätter:Yeah, I would say we should edit for that next year.
Stefan Starflinger:I also would like to say that right now, we still don't have 3 clients that are consistently working and not forking off.
Stefan Starflinger:So I still managed to get Nethermine and Geth to fork off, and kind of the con… Or the prerequisite for us to, launch the DevNet properly, so that clients
Stefan Starflinger:can work with a little bit of a transaction load, especially I would like to remind everyone to try out the EVM buzz.
Stefan Starflinger:scenario, the spammer, and see that the client, is stable with it. Red, seems to be working well, and Diesel seems to be working well so far with it. But otherwise, the other clients are still struggling.
Toni Wahrstätter:Perfect. Thanks a lot for that update, Stefan. Then I would say, let's… yeah, is there anything else someone wants to bring up?
Toni Wahrstätter:Otherwise, we can take the discussion offline.
Toni Wahrstätter:Awesome.
Toni Wahrstätter:Then, thanks everyone, and see you at the next Waco Calling 2 weeks.
Jared Wasinger:Yup, bye-bye.
Chat Logs
00:06:19
Stefan Starflinger:BAL is part of the block it’s required right?
00:06:55
Jared Wasinger:It’s required to follow the chain at the tip
00:07:03
Jared Wasinger:But it will be served by the CL over the engine api
00:07:47
Stefan Starflinger:What about providing them via RPC?
00:08:04
Stefan Starflinger:The balance changes are interesting for users
00:08:22
draganrakita:If they are optional in p2p, that would work
00:08:51
Stefan Starflinger:Is full sync still supported by clients as a fallback?
00:10:32
Jared Wasinger:yes
00:10:59
Ben Adams:full sync you can reconstruct the bals and then compare to the hash
00:11:17
Stefan Starflinger:Sounds very expensive
00:11:37
Jared Wasinger:I haven’t looked into them yet
00:11:46
Ben Adams:you contruct the bal anyway to verify
00:11:52
draganrakita:Same here, didn’t look, will send it to team
00:12:02
Karim T.:Replying to "full sync you can re..."
Yes but it’s very long
00:12:20
draganrakita:I will be fine without metrics in devnet0
00:13:23
Stefan Starflinger:Replying to "full sync you can re…"
Yeah the receiving client sure but then also the sending client needs to recreate them
00:13:26
Łukasz Rozmej:Debug logs with diffs only!
00:13:42
Ben Adams:Diffs rather than hash mismatch
00:15:50
Karim T.:What we do if the order is different.
00:15:56
Karim T.:But content is the same
00:16:18
Karim T.:For the hash
00:17:14
Karim T.:👍
00:18:01
Stefan Starflinger:Any updates on including generatedBlockAccessList in the debug_badBlock call?
00:22:21
Stefan Starflinger:👍🏼
00:22:28
Stefan Starflinger:And nethermind?
00:25:29
devconnect:Yes I will look into the remaining fails this week
00:25:37
devconnect:(felipe)
00:28:39
draganrakita:I like idea that we have all io access be bounded by size of the block
00:30:13
Karim T.:Only adding the read of the transaction that are exceeding a certain amount of SLOAD
00:44:29
Ben Adams:Is at EthClient Summit
00:48:21
Karim T.:Bloat devnet with BAL
00:49:15
Stefan Starflinger:Shouldn’t it be possible to get this kind of insight with kurtosis ethereum package and if not what is missing?
00:51:48
Jared Wasinger:Should be straightforward for geth
00:52:37
Karim T.:Not difficult for Besu
00:56:15
Karim T.:For the moment we don’t have yet prefetching of the read from BAL. Do we want that for the next devnet or later ?
Summary
9 highlights
· 3 action itemsExperimental
Summary
9 highlights · 3 action itemsExperimentalsyncing and storage
testing and debugging
state locations debate
- Read state locations add 50% to BAL size; benefit unclear at high gas limits00:25:29
- Proposal: Include read transaction indices only for first occurrence; reduces duplication00:36:09
- Dragan: Knowing what to read matters more than read order for parallel I/O00:44:29
- Nethermind team testing 500M-1G gas blocks by combining mainnet payloads00:49:15
Decisions
Action Items
Targets
- Devnet0 - State location prefetch flag and benchmarking capability needed00:58:35