Toni Wahrstaetter:Perfect. Hello, everyone, and welcome to the sixth breakout call of EAP7928. It's November the 5th.
Transcript
Toni Wahrstaetter:And let me directly post the agenda into the chat.
Toni Wahrstaetter:So that everyone can have a look. We have quite a few topics, because I put all the spec changes we made in the… since the last breakout call into…
Toni Wahrstaetter:Into the agenda today, just so that people are aware of what happened since last time.
Toni Wahrstaetter:This basically includes fixing how we track withdrawals, so you might remember in the last breakout call, we discussed how withdrawals are handled, and we had some bugs there in the specs.
Toni Wahrstaetter:Besides… Besides that… Yeah, there weren't…
Toni Wahrstaetter:any other major changes, at least as far as I'm aware. Maybe, Felipe or Rahul
Toni Wahrstaetter:If you want to add something to that, feel free.
raxhvl:I think that covers it, Tony.
Toni Wahrstaetter:Perfect.
Toni Wahrstaetter:Awesome.
Toni Wahrstaetter:Yeah, have a look at the agenda. I linked all the specs, and besides that, you can see the agenda point proposed changes, and this is then directly the first topic that we should discuss today.
Toni Wahrstaetter:Namely, do we want to include gas used values of each transaction?
Toni Wahrstaetter:in the block lab access list.
Toni Wahrstaetter:And… I've already spec'd it, so you can look at the specs, how it would look like.
Toni Wahrstaetter:And the main motivation behind this change was allowing clients to better parallelize transactions and avoiding
Toni Wahrstaetter:Worst cases that originate from not being able to distribute transactions.
Toni Wahrstaetter:Evenly across your course.
Toni Wahrstaetter:So, I'm not even sure if this is the right,
Toni Wahrstaetter:mental model I have on that one, so I'm curious what client teams think about this one.
Toni Wahrstaetter:If the gas used is actually something that would be helpful, or… If you think, like.
Toni Wahrstaetter:Just naively, parallelizing them in the sequence, how they are in the block.
Toni Wahrstaetter:Is… is good enough, or… Might even, yield the same results.
Toni Wahrstaetter:So if anyone has thoughts on this one…
Toni Wahrstaetter:I would just… I would assume that the default is just we don't do it.
Toni Wahrstaetter:But I think we should consider it, at least thinking of, Certain worst-case scenarios where…
Toni Wahrstaetter:You could end up with, with big transactions, being executed on one core, while other cores might idle.
Gary Rong:So, personalizing it's really hard to say that how much performance gain we can have by adding these,
Gary Rong:inductor. One good way is we can try to compare the So, execution time.
Gary Rong:With this, gas used, indicated or without.
Gary Rong:So that,
Gary Rong:We can, like, try to measure the real, blog, and, then we can decide if we want to include it or not.
Toni Wahrstaetter:Yeah, good point, and maybe just one question I would have regarding that. What would actually change if I give you a list of the gas used values? Would you just,
Toni Wahrstaetter:Modify the order in which you present transactions to the exec… to the executor.
Toni Wahrstaetter:Or is there a process where you're really like, okay, I have transaction 3, 5, and 7, and they are all very big, so I make sure I distribute them across core 1, 2, and 3, and make sure they are not landing on the same core? Is there even something like this happening?
Gary Rong:I hadn't really, like, thought about it, but I guess,
Gary Rong:We will try to, schedule the heavy transaction first, so that they can be… there's no waiting time, and we can try to…
Gary Rong:Give it, like, more execution time.
Gary Rong:Yeah, it sounds like a very naive approach, like, we schedule the heavy transition first, and then…
Gary Rong:try to emit the transaction one by one to different threads.
Gary Rong:Yeah, the… I don't have, like… Any…
Gary Rong:like, a useful comment for it, because I never…
Gary Rong:I, like, haven't did any experiment about this.
Toni Wahrstaetter:Yeah, Dragon?
draganrakita:I think it's not worth it. Rough math around that, the worst case, if…
draganrakita:If the stealing of the, like, scheduling of transactions to the course is done correctly.
draganrakita:The worst case can potentially be… because we have, like, 60 million cap limits, 60 million gas per transaction.
draganrakita:The difference is, like, 50% faster execution when everything is parallel.
draganrakita:With both, that the speedup is, like, 3-4 times, 50% doesn't matter at all.
draganrakita:So it doesn't, if the workstealing is done correctly, I don't think it's worth adding that.
Toni Wahrstaetter:Yeah, I see the SEO point.
Toni Wahrstaetter:Yeah, the example, a more tangible example, might be assuming 60 million guests, and we have, like, or a little bit above 60 million, we have 4.
Toni Wahrstaetter:let's say we have one max size transaction, and all the available… the remaining gas is consumed by EVE transfers, for example.
Toni Wahrstaetter:What you would want to avoid is executing all the EVE transfers.
Toni Wahrstaetter:before the big transaction. So what you want to do is parabolizing them, I guess?
Toni Wahrstaetter:Yeah, but I can see your argument that it might just not be worth it, and just confirming also what Gary said. Go ahead.
draganrakita:In… in general, at least, I still didn't… I'm not sure even if I'm going to implement that, but you want to implement something like steel intersection. You have 4 threads, or 4 cores, that execute in the… in the parallel, and they steal for one… one single queue.
draganrakita:And whatever the core finish, executed transaction, it steals another transaction from the queue.
draganrakita:In that sense, every core that's not… that wants more work will steal the task from that queue.
draganrakita:At least, this is, like, one of examples how could this, like, work.
draganrakita:Worst case for that, if the last Task is the biggest one.
draganrakita:And the biggest task is 60 million, so it's like… It's… yeah.
draganrakita:It's not worth it, to be honest.
Toni Wahrstaetter:So compared to, like, 60 million gas limit, it's already, like, one-fourth.
Toni Wahrstaetter:And compared to, like, 100 million, like, the effect will just become… Smaller and smaller.
draganrakita:Yeah, exactly.
Toni Wahrstaetter:Yeah, anyone else who wants to chime in, voice an opinion?
Toni Wahrstaetter:Otherwise,
Toni Wahrstaetter:We don't have to take a decision today anyways. The default would be just we don't change anything, and the gas used would stay out of the block of access list.
Toni Wahrstaetter:We can also use, the time around DevConnect to discuss that further. Maybe clients, get to comparing how they would eventually parallelize
Toni Wahrstaetter:But in general, I agree. This argument of Dragon does make a lot of sense, that
Toni Wahrstaetter:We might improve the worst case.
Toni Wahrstaetter:But it might not be worth it.
Po:hello, Tani. Yeah, go ahead. I have a question. Because I joined later, the question is, shall we schedule the transaction execution by guest used?
Toni Wahrstaetter:Yes, exactly. If we put the gas used values into the block lab access list, if this is kind of useful information for parallelizing.
Po:Okay, I recently just did an experiment with gas limit. Yeah, it seems this information is
Po:Enough, and it seems the most time-consuming
Po:Transactions will dominate the block execution time.
Po:Yeah, maybe later I can show a demo, so you can see the results.
Po:Yeah, we can't schedule just by the gas limit, yeah.
Toni Wahrstaetter:Exactly, yeah, this… these are, like, the… there are three options, scheduling just by… or four options. Just naively scheduling, by their order, as they are in the block, then you can use the guest limit.
Toni Wahrstaetter:then you can use the gas used, which would require us to add it into the block of access list, and the most hacky solution would be you take the balanced diff, and from that you infer the gas used, because
Toni Wahrstaetter:Every, every time you pay fees, it's also a balanced diff.
Toni Wahrstaetter:yeah, it might be attackable, and the cleanest solution would be to just use the gas used, but as Dragon said,
Toni Wahrstaetter:Worst case scenarios are still, like, very contained.
mark:Bye.
Toni Wahrstaetter:the 7825 transaction cost limit.
Toni Wahrstaetter:And we might just ignore it, ignore the whole topic, and don't include it.
Toni Wahrstaetter:But yeah, definitely happy to look at your results there.
mark:Yeah, so sorry, just, I joined a bit late, so I'm not… So, one of the things that we've discovered with parallelization in Aragon is we've basically ended up with two phases. We do the execution phase.
mark:And then we do the receipt phase at the end, because…
mark:Depending on the order in which parallel execution is happening, if you use a common gas port, it basically, you're gonna go out of gas.
mark:Essentially, when, in fact, you wouldn't, once you… once you get to the end of it. So having… having the gas per execution would make that process a lot easier, basically.
mark:Because it means while you're execute… you know, when you execute a particular transaction, you can check its gas at the time.
mark:Which you generally can't do when you're executing in parallel.
mark:Because you don't know the order.
mark:If you see what I mean.
Toni Wahrstaetter:Yep.
Toni Wahrstaetter:Yeah, this is… this is… this is exactly the argument. Yeah.
mark:I had too, yeah.
Toni Wahrstaetter:Dragon's counterargument is, like, that even if it turns out you scheduled them in a way where you did, like, the worst job possible, for example, you were tricked into scheduling that way, then the worst case would still be, like, you would maximally have, like, one transaction on top.
Toni Wahrstaetter:While all cores are idling.
Toni Wahrstaetter:And this transaction only has a size of 16 million, which is, like, 1 fourth of…
Toni Wahrstaetter:Yeah, what do you mean?
mark:The problem is more that if you've got transactions with reverts in the middle of them.
mark:As you're processing, the transaction looks like it's using more gas than it should do, and…
mark:What then happens is, it's generally fine, but if you're on a full block.
mark:And you get to the final transactions, you can basically run out of gas too soon.
mark:If you see what I mean.
mark:So you get a set of edge cases around… Failing blocks, on gas.
mark:When actually they don't, because the… effectively, the transaction looks like it's using a lot of gas, then it reverts.
mark:So…
draganrakita:I'm not exactly following this use case.
draganrakita:Are we talking about if the intersection is not spending exact amount of gas that it should be?
draganrakita:Or something else.
mark:Well, I mean, the transient gas cost of a transaction
mark:Changes, depending on whether it reverts or not.
mark:And…
mark:Basically, that means you end up with some cases when it looks like they're overusing gas. So you get out of gas sooner than you expect.
mark:If you know what the transaction gas is.
mark:You're never gonna hit that situation.
draganrakita:You mean if the trace section reverts, all the gas are…
draganrakita:He spent, even if the gas is not, like… used.
draganrakita:It's it like that.
mark:Yeah, it looks like it's… so it depends, to some extent… extent, it depends how you're doing the gas calculation.
draganrakita:Yeah, then this is, like…
draganrakita:Good case where the cycle should take a lot more time, but in reality, it takes one end of the
draganrakita:time than it should be. It's like, yeah, it's faster to execute than we expect from the gas.
draganrakita:Okay, that's a good comment.
mark:Yeah.
Jared Wasinger:Why is it's not clear…
mark:Does this…
Jared Wasinger:Sorry, go ahead.
mark:It just executes differently, and basically what we've discovered in testing is you get the occasional block.
mark:that you go out of gas right close to the end, basically. And it's only… so it only impacts full blocks, basically.
mark:Or foolish blocks.
Jared Wasinger:Yeah, I… I don't fully follow, but I… just to be clear, like, the implication of this…
Jared Wasinger:Of not having the gas used here is basically that if we…
Jared Wasinger:That we can't invalidate a block for the actual gas use of all the transactions, going over what is allowed by the gas pool.
Jared Wasinger:We can't do that until we, like, we have to execute all the transactions and validate it at the end, right?
mark:Yeah. In the collection of the receipts? Is that… that's the issue you're getting at here? Yeah, yeah, that's the issue.
Jared Wasinger:Okay.
Toni Wahrstaetter:Let's see.
Toni Wahrstaetter:So, essentially, at the very end, you would go through all the transactions again, and determine if they actually
Toni Wahrstaetter:Fit into the block.
mark:Yeah, which, you know, in practice, it's not a very computationally heavy process, so it doesn't make much difference to the… because… because you need… the thing is, you need to do that for the receipts anyway, because you've got a… you need the… you need the aggregate number as the… so you end up having to total the receipts at the end anyway.
mark:And comparatively, it's very little processing to do that, so you don't get a big advantage in doing that.
mark:Well, doing that whole process at the end doesn't seem to cost much time, in terms of the scheme of processing a block.
Toni Wahrstaetter:Right, and then…
Jared Wasinger:The main downside is… is just that you can't… you can't…
Jared Wasinger:You can't know that the block is bad until you've processed all the…
Jared Wasinger:Yeah, correct. transactions in this circumstance, so…
Toni Wahrstaetter:Right, but this is not really a DOS vector or anything, so… to me, it feels like…
Toni Wahrstaetter:This is not really a strong argument for having the gas used values in the popular excess list. I do see why it makes this process cleaner.
Toni Wahrstaetter:Because you could just compare each transaction individually and independently to the gas used value in the popular access list.
mark:Yeah.
Toni Wahrstaetter:Yeah, so it's a nice… Yeah, good point.
mark:I'm just… yeah, so, in the scheme of things, it's a nice path.
Toni Wahrstaetter:Right, yeah. Yeah, I agree to that.
Toni Wahrstaetter:Yeah, I think… I think for now,
Toni Wahrstaetter:We would… we would not include it, but, yeah, we still have some time to recite this topic. We don't have to take a decision of,
Toni Wahrstaetter:Today, So, we can also discuss, during DevConnect and at the next, breakout call again.
Toni Wahrstaetter:Yeah, it's good that we have,
Toni Wahrstaetter:Brought this topic up so that people are aware of this possibility.
Toni Wahrstaetter:Any more comments on that?
Toni Wahrstaetter:Perfect, then let's move on to the next agenda item.
Toni Wahrstaetter:Actually, the next agenda item is client, indefinite updates.
Toni Wahrstaetter:So, if you want, Stefan, I would invite you to Give us an update there.
Stefan Starflinger:So, we, overhauled a little bit how we do definites. We added a new boot node, setup, so that's why we've been kind of iterating on our side a little bit.
Stefan Starflinger:I can also share, kind of, the spec updates,
Stefan Starflinger:that I haven't mentioned yet. PK has, added, EVM fuzz.
Stefan Starflinger:Tool to the spammer that we can use.
Stefan Starflinger:It basically creates contracts with, different opcodes.
Stefan Starflinger:And if you run that with different client pairings, there's still a few issues that can be found. So I'm generally kind of waiting for a very or relatively stable kurtosis testnet.
Stefan Starflinger:And to see that all clients are decently working together.
Stefan Starflinger:And I think also with the spec changes, it would be good to get some client updates also.
Stefan Starflinger:kind of where we are at, because it's, it's, wouldn't make sense to spend money on a DevNet, if it's not running stable yet.
Toni Wahrstaetter:Yeah, thanks for the update. Maybe let's ask client teams how…
Toni Wahrstaetter:Far in the process are we, or how close do you feel being able to turn a definite?
Toni Wahrstaetter:When it would make sense, of course.
Jared Wasinger:Yeah, I guess I'll just go,
Jared Wasinger:I think that Geth is mostly ready, but, like, so we just had this new…
Jared Wasinger:test release… so, essentially, the Coinbase issue hasn't been entirely settled in Geth,
Jared Wasinger:But other than that, we would be ready to go.
Karim T.:On Bizzou, we are passing all of the tests related to Amsterdam, but we have a… I think, as other clients, we have a lot of issue in the tests that are testing old fork.
Karim T.:So, I'm not sure if we want to be sure that we are passing this test before starting the DevNet, or if we just say that it's bad configuration of the test, and we will see, maybe in the next version, if it's okay.
Karim T.:Because we don't really know if it's just because of a bad configuration in the test, or if it's really a…
Karim T.:A real issue or something, isn't it?
Stefan Starflinger:Do we have someone? Yeah, sorry.
Marc:I was just gonna say, for Nevermind, it's the same, just, going through these new tests,
Marc:This new test release, and just checking, why some of these tests are failing.
draganrakita:Same with RAT. I don't have any pending items on both sides.
draganrakita:So… Fixing tests on spec side, running them to confirm that everything is okay. It's, like, the next thing.
draganrakita:And after that, I think DevNet Zero could be spun up.
mark:So, where we are from an Aragon perspective is we've now got an implementation that we're testing. We're getting quite a few Hive test fails, so…
mark:Chome, who's actually working on it, is not on the call, because it's not good from a time zone perspective, but I'm wondering if there's somebody he can contact, just to find out whether
mark:Because this is the first time we're going through, he's trying to work out whether the fails are…
mark:us, or DevNet issues, testnet issues, or Hive issues. So, it would be good if there's somebody that we can put him in touch with.
mark:Because we're kind of bootstrapped in the hive process at the moment.
Toni Wahrstaetter:Yeah, definitely. So, I think the… the two people with the most insight on that process are Felipe and Rahul.
Toni Wahrstaetter:So, yeah, Rahul, I just see you on the call of the deep end, maybe you could get them a little bit?
felipe:Yeah, yeah, I'd like to catch up.
felipe:With everyone on the state of the tests, so…
felipe:There are a few spec changes that are needed that, actually impact…
felipe:So this is based on conversations on the block access list, Ether and D channel.
felipe:There's a more recent conversation with… with Sam, who's… Been leading the… the Execution Specs project.
felipe:And these changes are going to actually impact the specifications for all forks.
felipe:And so what I want to do right now is release a patch.
felipe:For the tests that has The changes that we need so that we can
felipe:Generate cleaner tests for block access lists. And this includes all the old forks.
felipe:So what this release did is it basically
felipe:Build all of the tests that we have in our test repo.
felipe:Across all of the forks.
felipe:generating block access lists for all of them, and so I expected quite a few failures.
felipe:Because this will reveal edge cases that we just haven't written targeted tests for block access lists yet.
felipe:But knowing that there are spec changes that are needed, means…
felipe:Right now, it's really hard to tell, which tests
felipe:Are, actually bad tests, and which ones are failing because the specs aren't up to date.
felipe:So there's this… Discrepancy of resolving
felipe:the specifications quick enough for Amsterdam, at least, so that we can
felipe:Put out a patch release for the tests, but also, ultimately, we have to go back and
felipe:Fix the specifications. It's basically related around charging gas.
felipe:For, memory expansion and… Loading account earlier, within each call.
felipe:And then continuing on with the call afterwards, so that we…
felipe:Only track the account if we have enough gas to track the account, and things like these.
felipe:And so, yeah, unfortunately, the current state of the tests don't… aren't very representative of
felipe:What the specifications, should be.
felipe:Currently.
Toni Wahrstaetter:Thank you very much for the update.
Toni Wahrstaetter:Yeah, I've just posted the PR I'm working on in the chat.
Toni Wahrstaetter:Just to summarize.
Toni Wahrstaetter:there are two things… some things are more like cleaning up. For example, putting the state tracker into the block environment instead of the state.
Toni Wahrstaetter:The more crucial thing you were referring to is really
Toni Wahrstaetter:Kind of splitting up the charging gas from checking the gas, and making sure that if we
Toni Wahrstaetter:Run out of gas.
Toni Wahrstaetter:or let me put it differently, that we never put an address into the block club access list, or any entry into the access list.
Toni Wahrstaetter:When we run out of gas afterwards.
Toni Wahrstaetter:Right, so you want to kinda… what, Dragon… I think Dragon was that… was saying that, or…
Toni Wahrstaetter:who was… I don't remember who brought it up, but is that already, like, addressed in my PR?
felipe:I… I'll need to go back. This is the new PR, right, after we've had conversations with Sam?
Toni Wahrstaetter:Right, yeah.
felipe:Okay, yeah, I had the… so I also posted my gas charge split PR, and this does, stabilize all the bad tests.
felipe:And this is… I'm not sure if this is all of the changes that we need, but I think this covers most of them.
felipe:And it does have all of the targeted block access list tests that are up to spec passing, and so I feel like
felipe:this PR that I worked on, and whether those changes are also in your PR, some version of that.
felipe:Is what we need for a more stable test release.
Toni Wahrstaetter:Perfect, yeah, then I will take those changes out of my PR and just leave it at this refactoring, where we move the, state tracker around.
felipe:Yeah, that sounds great. Internally, I think I just… I would like to get,
felipe:The steel team on board with
felipe:putting these changes up for Amsterdam first, and then we can focus on the full forks, refactor to how the specifications charge gas in general.
felipe:After the fact, so that we can…
felipe:Get our tests up to speed.
Toni Wahrstaetter:Perfect, thank you.
Toni Wahrstaetter:Awesome, yeah, I haven't, haven't seen your PI yet. I will, I will review it. Perfect.
Toni Wahrstaetter:Any other, updates we should discuss?
Toni Wahrstaetter:Regarding testing, regarding the specs?
Toni Wahrstaetter:Making sure we don't forget anything.
Toni Wahrstaetter:I'm relaying this message from Stefan. The clients please run the EVM fast scenario for Spammer.
Toni Wahrstaetter:Great.
Toni Wahrstaetter:If there's nothing else, then we can move on to the next agenda item.
Toni Wahrstaetter:Which is… A rather old topic, it's still about, do we want to include the reads, or… Don't.
Toni Wahrstaetter:With reads, we usually mean the state locations, so unmodified accounts that come with empty, lists of changes objects, and also
Toni Wahrstaetter:storage keys?
Toni Wahrstaetter:So everything that is touched within the block but not modified. We have discussed that topic, a while ago.
Toni Wahrstaetter:And Gary did some benchmarks very recently.
Toni Wahrstaetter:Gary, are you on the call? Do you wanna… Quickly.
Gary Rong:Yo.
Toni Wahrstaetter:Yeah, what you did there.
Gary Rong:Yeah, I can, share my… right up, can I share my screen?
Toni Wahrstaetter:Yeah, should work.
Gary Rong:Okay, can you see my screen?
Toni Wahrstaetter:I can see it, yeah.
Gary Rong:Yeah, so, I've been discussing with Tony recently about whether to include the read location in the BAL,
Gary Rong:So, at the beginning, I was at the opposite side, because I think it's not that easy to…
Gary Rong:properly gets all the relocation during the block's execution.
Gary Rong:And, because we need to compare the BAL data in the consistency,
Gary Rong:So if we provide, like, slightly different data, since the consensus will be few.
Gary Rong:like, I am not very confident about it, so I suggest that maybe we can,
Gary Rong:Exclude the location in the initial version.
Gary Rong:But, yeah, later I try to, measure the performance gain we can have by including… by utilizing this relocation.
Gary Rong:So, I think the intention of the BAI is to try to optimize the block's execute… block execution, and, so that we can push the network S limit, upwards safely.
Gary Rong:But we need to ensure that the BL can benefit, basically, all the potential scenarios in the block's execution.
Gary Rong:One potential worst case I can imagine is that the read-heavy block.
Gary Rong:And so now the… gas cost for the code as load is.
Gary Rong:$2,100, and for the account access, it's 2600. So, with the current network guest limit, we can, in theory, have, 20…
Gary Rong:thousand, S-loading within a single block.
Gary Rong:Yeah, so I try to measure, like, how much time we need to spend if we want to access this
Gary Rong:So many, storage or accounts.
Gary Rong:And, with Osaka, like, we limit the maximum gas limit within a single transaction to 16 million.
Gary Rong:So,
Gary Rong:It means that we can at least have 4 transactions within a single block. So the baseline I have chose is a thread with 4,
Gary Rong:So the first thing I did is, I tried to measure the batch rate with different thread over the raw SSD.
Gary Rong:And, yeah, it turns out that, with 34, we can achieve this, 70s, 7…
Gary Rong:70,000 QPS, curry per second, and with 128, we can achieve, Roughly… 10x the speed up.
Gary Rong:so, yeah. And,
Gary Rong:like, it's the real performance over the raw SSD, and the next thing I did is I tried to perform, this batch read over a real E3 million guest node.
Gary Rong:The background is that, the database size is, 288 gigabytes for the… Key-value store.
Gary Rong:And, yeah, so… With 4thread, so… QPS is around… Mmm, 22K.
Gary Rong:And,
Gary Rong:if we give, like, 128 thread, the QPS can go up to, 200K, so it's 10x faster.
Gary Rong:And this number, like, totally changed my mind, and I think it's worthwhile, because,
Gary Rong:It is 10x faster for this particular, rehabic scenario.
Gary Rong:And another interesting thing is,
Gary Rong:I've… I have observed is that, before the thread, 32, if we give more thread, like, higher concurrency, the reperformance, will be increased linearly, but after that, we, like.
Gary Rong:Hit the acceleration point, and the performance… there is no significant performance gain by having more concurrency.
Gary Rong:So I think it might be relevant with the physical structure of the SSD, because internally it has some…
Gary Rong:Channel inside, and it has a specific capacity.
Gary Rong:Yeah, so let's say the acceleration point is 32 thread.
Gary Rong:And, given that, for a single transaction, the maximum gas limit is 16 million gas. So, I think we can say that
Gary Rong:Before the network gas limit, be increased to 500 mega… a million gas.
Gary Rong:So BAL waste red location will still be beneficial for us.
Gary Rong:Yeah, but all the benchmarks are performed over the gas implementation, like, different client can… because different clients use totally different database, and the number might be slightly different, or could be completely different, so I would recommend you guys to
Gary Rong:simulated the same thing, and to make sure, like, our numbers are aligned with each other, so that we can make, so…
Gary Rong:Final decision to include it or not.
Gary Rong:Yeah,
Gary Rong:I guess that's it. I did link my benchmark program here, so if you want to have a reference implementation, you can check it out.
Gary Rong:Yeah, but otherwise, syncs, that's it.
Toni Wahrstaetter:Awesome, thank you very much.
Toni Wahrstaetter:Are there any questions people have?
Łukasz Rozmej:So, one comment from me, because Gary used 28 thread CPU, so this 32…
Łukasz Rozmej:thread limit, increases might be as well from the CPU being, completely used, but I don't know. That might be a good thing to, check.
Łukasz Rozmej:So, maybe on different CPU and different disk, or, like, especially, like, disk in RAID, we could achieve even more, scaling.
Gary Rong:Yep.
Gary Rong:It could be possible, so, yeah, we can try to use a different setup to run the benchmark again, and .
mark:Sorry, so how many CPU threads… so how many CPU threads have you got?
mark:Cause you, my…
Gary Rong:In my setup, I use, Intro, like, 147… K.
Gary Rong:So it should have, like, roughly 20s.
Gary Rong:cost for… concurrency.
mark:Yeah, so the question…
Łukasz Rozmej:20 cars and 28 threads.
Gary Rong:Hmm.
mark:But, so, for Disgao.
mark:you're not… it may be offloaded off the core. The interesting question is, if you have fewer cores and the same number of threads, what does it do to the performance?
Gary Rong:Yeah, I think,
Gary Rong:When I tried… when I did my benchmark, like, it is the only program I am running, so I would assume that it can use all the, available cores.
Gary Rong:But, as I said,
Gary Rong:in my setup, like, I use this particular CPU and this particular SSD. If we use something different, maybe the number could be, like, slightly different. So, yeah.
Gary Rong:Yeah, I'm just wondering how much CPUs were doing.
mark:Because they're basically.
mark:this guy,
mark:or… or device I have.
Gary Rong:So in this, benchmark case,
Gary Rong:I… basically, we have no, CPU, computation.
mark:Because, we assume it's a really heavy…
Gary Rong:block, and the read is the slowest operation. So, for the worst case, it's just all the read without any VMIC computation.
mark:Yes, that's what I'm saying. So that implies that, actually.
mark:you can… your number of threads is not CPU dependent.
mark:It's… you should be able to increase the number of
mark:threads you've got allocated to I.O.
mark:Because… the CPUs are just idle. They're probably waiting for the I.O. to return.
mark:Because it's the I.O. subsystem that you're loading.
mark:If you see what I mean.
mark:Anyway, so it'd be interesting to see what the numbers would look like on less threads.
mark:I mean, less CPUs, same number of threads.
Gary Rong:Yeah.
Toni Wahrstaetter:Yeah, and it would be great if clients can also reproduce the… reproduce that, individually, just if we have.
Toni Wahrstaetter:more data points on the topic, as well as we should try it with double the state size. So currently, in Gary's analysis, the state was at 288 gigabytes.
Toni Wahrstaetter:I think right now… We have a state growth of, like, 60… 6200 gigabytes per year.
Toni Wahrstaetter:So, we should definitely try it with double the state size, too.
Toni Wahrstaetter:And see if this changes something in the numbers.
Toni Wahrstaetter:But in general, because one thing we have to consider when it comes to the state breeds is that the worst case
Toni Wahrstaetter:Block level access this side.
Toni Wahrstaetter:Size is also for the exact same scenario.
Toni Wahrstaetter:So, we might end up in a situation where we have, like, the worst-case block containing a lot of storage reads, while a rather big block… it's not the worst-case block, because call data will still be
Toni Wahrstaetter:Allowing you to get to a bigger block, but it's a… it's a big block. So, for example, at 60 million guests, the blocks of access list would be at, like.
Toni Wahrstaetter:850 kilobytes to 1MB.
Toni Wahrstaetter:Which takes a… with 50 ambit, takes 165 milliseconds to download.
Toni Wahrstaetter:So this is all something, all things we have to consider here. So if we, like, get something from the state locations, then it must be significant, and it must be
Toni Wahrstaetter:Worth the costs we pay, on bandwidth.
Toni Wahrstaetter:But based on this analysis, it looks like it.
mark:And it looks like it's… it's…
Toni Wahrstaetter:tops at 500 million. So, like, at 500 million, we have enough transactions
Toni Wahrstaetter:That you can, nicely parallelize them.
Toni Wahrstaetter:And then… The benefits we get are a little lower, or less significant.
Toni Wahrstaetter:And… Yeah. Go ahead, Mark. You wanna say something?
mark:I was going to say, the other… the other thing, though, is that if you're running… when you start running parallel execution, if you parallelize the transactions.
mark:You get a fair amount of natural Parallelized reads, anyway.
mark:Right, because if you've got all… if you've got all your executions running.
mark:N of them hit a read, so you're running reads in parallel anyway.
mark:If you see what I mean.
Toni Wahrstaetter:Exactly, exactly. That's why Gary took 4 as the benchmark, because essentially, with 60 million transactions.
Toni Wahrstaetter:You will be able to parallelize 4 transactions at the same time.
Toni Wahrstaetter:Which is essentially the same as doing, like, Using 4 threads for reading.
Toni Wahrstaetter:Right? This is what you're referring to.
mark:Oh, yeah, and I'm saying, basically, so, if you've got all the… I mean, in practice, if you've got all the reads up front when you start parallel execution, you'll kick off a set of parallel reads, but also, you've got, like, a race condition in the sense that
mark:you won't… I don't think you'll wait to start executing transactions, and then, kind of, the early transactions will actually go to the disks themselves.
mark:Right? Because they get there before the read comes back.
mark:And the question, if you care about bandwidth, is what boost do you get over the natural parallel flow, if you see what I mean?
mark:Because you could be driving the disk quite fast anyway, just from the parallel execution process.
mark:So you might not need… you might not need the read list, because the transactions will get there pretty quickly anyway, if you see what I mean.
Toni Wahrstaetter:Yeah. Yeah, that's a good point, and that's, I would say, exactly the reason why we have to consider if reads actually give us
Toni Wahrstaetter:enough to be worth the costs we pay. Right now, based off Gary's analysis, it looks like it is worth it, because
Toni Wahrstaetter:It was, like, 8x or something, and in absolute numbers, a few seconds.
Toni Wahrstaetter:Difference when we are at 500 milliseconds? Or a few, tenth of a second difference?
Toni Wahrstaetter:if I remember correctly, like, a little more than half a second of difference, and then it looks like it's worth the bandwidth costs we pay. But yeah, I think… I think it's very close, and we should definitely get some numbers on
Toni Wahrstaetter:Different clients, different implementations of the simulations, and second, on a larger state.
Toni Wahrstaetter:And, yeah, maybe we can then take a… take a decision.
Toni Wahrstaetter:As soon as we hefted.
Toni Wahrstaetter:I just see Dragon in the chat. There is EAP7870, if I remember correctly.
Toni Wahrstaetter:That defines, like, minimum hardware specs, ELP.
Toni Wahrstaetter:7870, am I correct? Yes.
Toni Wahrstaetter:I will post it into the chat.
Łukasz Rozmej:Can we just implement both and run real-world, like, testing later, and decide closer to the deadline?
Łukasz Rozmej:on that.
Toni Wahrstaetter:If you're happy.
Łukasz Rozmej:Being measuring both, right, having both access lists with and without, or, like, maybe distribute with, but then measure execution with and without, right? On one machine like that, one machine like that, and compare
Łukasz Rozmej:The actual results in practical blocks, and worst-case blocks, too.
Gary Rong:Yeah, it is, like, the intention of makes this benchmark, because we can… Like, have a…
Gary Rong:Feelings at how much performance difference we will have.
Gary Rong:So,
Gary Rong:Like, if we run the BAL in the real, scenario, it would be perfect, but at this stage, the early stage, we can try to measure the performance difference with this benchmark first.
Gary Rong:It's what I'm trying to say.
mark:Yeah.
mark:But I kind of agree that we need to go through… we should leave it… so I think leaving it in at the moment is a good idea, and they… we ought to leave it in for quite a long way through the…
mark:the DevNet process, because once we've established some operating parameters, what we actually want to do is take it out and see whether it makes any difference.
mark:If you see what I mean. Because otherwise, you really don't get the… the… what does it do to the overall end-to-end parallel flow?
mark:Because one of the things that I've discovered
mark:testing parallelization in Aragon is, you really need to have all the pieces together.
mark:Because they… they have different behavior when you test them individually.
mark:Basically, in the end.
Łukasz Rozmej:So, if we have them in for now, we don't even have to take them out, we just have to have a second implementation that doesn't use it, and we can compare it live, the difference on…
mark:Yeah, yeah, correct, yeah, yeah. Yeah, I agree with that.
mark:Because you can say, well, what practical difference does this have over your block time if you use it or you don't?
Toni Wahrstaetter:Yeah, I can see that argument. The only thing I was,
Toni Wahrstaetter:Hesitant about was taking a late decision on that, because, of course, we would love to… to have the numbers today instead of tomorrow.
mark:Yeah.
Toni Wahrstaetter:So I don't wanna… you know, I'm not sure if it's the cleanest approach to have, like, two releases running at the same time, or if we…
Toni Wahrstaetter:Just do the benchmarks now, and make sure we get it… we get to as close, as realistic for a south as possible.
Toni Wahrstaetter:I mean, from the spec side, it wouldn't be a big problem, but I would assume that in client implementations, it's not just taking them out versus leaving them in.
Łukasz Rozmej:Not really, it should be relatively easy to do, because you could just have a parameter if you load those, in the background thread, based on this access list, or you don't, or you just ignore the street access list, right? In a way.
Toni Wahrstaetter:Yeah.
Łukasz Rozmej:So, I…
Toni Wahrstaetter:So there are some edge cases attached, right? For example, if you revert and you have written to something in the specs, for example, we have this custom function that converts a write to a read for reverted writes.
Toni Wahrstaetter:stuff like this would then be needed to be commented out, but I agree, it should be possible to…
Toni Wahrstaetter:Yeah.
Toni Wahrstaetter:To do so.
Gary Rong:Yeah, and another point is that
Gary Rong:Personally, I think there are so many HGKs in, like, collecting the relocation.
Gary Rong:If we can somehow make an early decision that we don't want this relocation, it would be much easier for the implementation.
Toni Wahrstaetter:Right, yeah.
Toni Wahrstaetter:Yeah, the good thing is that we have… we have them in the EAP right now, and clients are basically ready to… to ship them, or on the track to ship them.
Toni Wahrstaetter:Compared to adding them later on, which might be… Way too difficult.
Toni Wahrstaetter:So, I… I would suggest we keep them in for now, but it's great that we now have more data points on that topic.
Toni Wahrstaetter:And then it would be great if we can simulate it even more. Maybe we get some clarity before we even need to run multiple versions on a defnet and compare them empirically.
Toni Wahrstaetter:But yeah, let's see.
Toni Wahrstaetter:Let's keep the discussion on that, ongoing.
Toni Wahrstaetter:And keep the thread open.
Toni Wahrstaetter:Is there… Any… anything else we should discuss regarding, this optimization.
mark:No, I just want… I mean, it's… the summary of the discussion seems to be that benchmarks doesn't rule it out, right? It says, actually.
mark:is potentially going to give a big enough performance improvement to… that we should continue with it. That was what I was hearing.
mark:Gary's saying, I think.
Toni Wahrstaetter:Yeah, and Gary's benchmarks were also, like.
Toni Wahrstaetter:not super clear on what, what is the actual, to-do from them, right? Because they give us, like, an 8 to 10X,
Toni Wahrstaetter:With diminishing value, with increasing, block gas limit?
Toni Wahrstaetter:But at the same time, ignoring, increasing state.
Toni Wahrstaetter:So, I think… I think it's very difficult to…
Toni Wahrstaetter:To tell if it's worth it or not, just based on that.
Toni Wahrstaetter:But it's definitely a first data point we can anchor, anchor on.
Łukasz Rozmej:Interesting conclusion for implementation is that if we do have read access lists, we probably want to spawn, as many threads for transaction processing up to the num…
Łukasz Rozmej:we have on our CPU, but if we don't have read access lists, we probably want more threads on transaction processing than we have
Łukasz Rozmej:it will be stalled. So that's an interesting point for the implementation.
mark:Yeah, well, I think the reason I was asking about how much CPU has it got, the problem is, if you dedicate CPU to reading, you're not dedicating CPU to, computing transactions.
mark:So there's going to be a trade-off there, potentially.
Gary Rong:Yeah, but, I mean, like, in this particular scenario…
Toni Wahrstaetter:Same for the block… for the post-stage route, I guess, right?
mark:Yeah.
Toni Wahrstaetter:Perfect, yeah, thanks a lot, Gary, for the benchmarks. This is super helpful.
Jared Wasinger:Gary? Yeah, go ahead.
Jared Wasinger:Gary, you were starting to say something, and I…
Gary Rong:Can you hear me?
Jared Wasinger:Yep.
Jared Wasinger:Yeah, I can hear you.
Gary Rong:Totally.
Gary Rong:So what I'm trying to say is that, if we think that the read is, underpriced.
Gary Rong:So the worst case will be a block with all the, as load, with basically no EVM computation.
Gary Rong:So in this case, if we dedicate all the CPU cores to the S load, it should be fair.
Gary Rong:Because there is no, other EVM, computation, and, EVM computation is, like,
Gary Rong:Way more faster than the discrete.
Toni Wahrstaetter:Is that… is that realistic? Is it even possible to do, like, only S-loads without any computation in between?
Gary Rong:I'm not so sure, but I guess, it might be feasible to have some specific program to
Gary Rong:to do the, for example, external code hash for a non-existent account, and this address could be generated, like, randomly. In this particular scenario, it will basically touch all the pieces of the database, so we can avoid the data locality in the cache.
Gary Rong:It should be possible to correct such block to slow down the block execution, even with the current mainnet.
Toni Wahrstaetter:Thank you very much, yeah. Is there any… anything else on that topic?
Toni Wahrstaetter:Otherwise, let's get to the last point in the agenda, namely, in person.
Toni Wahrstaetter:meet up at DevConnect, so we have space at DevConnect that we can meet up and discuss, everything around block access list, progress.
Toni Wahrstaetter:There is no… Daytime yet, so this is still something,
Toni Wahrstaetter:I will work on, but I will keep…
Toni Wahrstaetter:everyone updated, and also make sure that people that are not at DevConnect will be able to join via Zoom, or…
Toni Wahrstaetter:We can meet Torin, or something.
Toni Wahrstaetter:But yeah, just as a heads up, we might meet and discuss things like the gas limit, the reads, and so on, at DevConnect.
Toni Wahrstaetter:Perfect! Before we end the call, is there any open topic we should discuss? Anything else we forgot that is, like, important until the next breakout call in two weeks?
felipe:Just on what…
Stefan Starflinger:Sorry, just on what Philippe said, there are, like, the debugging, there's just the hash if there's a mismatch between the bars and the logs.
Stefan Starflinger:But there's a debug endpoint that Bezos implemented, maybe other clients might want to look into implementing that, where they have the bad block that they disagree with, and they have the access list that they generated, so it's easy to compare.
Toni Wahrstaetter:Good point, yeah. Thanks, Stefan, for bringing that up.
felipe:Yeah, that's a nice second step, too, though. I feel like
felipe:because we have so many hive failures, if we had this in the debug logs, it would be…
felipe:Quite quick to just have them to compare against each other.
felipe:Something Mario, from Steel brought up is…
felipe:It would… it would be good for a future like this if we had, tracing aligned, too, at some point. I mean, this is somewhat in the future, but how we do EVM tracing, but we can trace what the bow is doing for each client.
felipe:But… this debug,
felipe:debug log of the full valve, on errors would be… would be quite nice, I think.
mark:Yeah, having spent the last…
mark:several months debugging parallel flows, that would be definitely useful, because you end up needing… what you want is one that worked and one that didn't, and to do a compare of them.
mark:Otherwise, it's really difficult to work out what the hell's going on.
felipe:Yeah, exactly.
mark:Yeah, so this endpoint, does it have a… has it got a specification?
mark:Is there something we can copy so we don't do something similar but different?
mark:The basic one you mentioned.
Stefan Starflinger:I don't think it has a specification. It basically just dumps the block as a top-level key, and under it, it's the generated block access list as another key, and that's pretty much it.
mark:Right, so it's basically just the block, and so, in the response, it's the block… And the bow, basically.
Stefan Starflinger:basically, I can share, in the channel, in the research channel, I can share an example as well.
mark:Yeah, no, that would be great, just so that we don't… because we might as well just copy it, so we're producing the same thing.
Stefan Starflinger:Yeah, and in the spec I shared above, I also shared the curl request to get it.
Stefan Starflinger:I'll also share an example there, yeah.
mark:Yeah, no, that would be brilliant, just so that… because we'll definitely implement it, because I think we need it, basically.
Stefan Starflinger:Sounds good.
mark:Yep.
Toni Wahrstaetter:Perfect.
Toni Wahrstaetter:Awesome.
Toni Wahrstaetter:Then we're at the end of the call anyway.
Toni Wahrstaetter:Thanks, everyone, for attending, and see you at the next one in two weeks.
mark:Okay.
felipe:Oh, thanks.
Gary Rong:U.S.
Jared Wasinger:Yeah, bail.
Chat Logs
00:03:19
Toni Wahrstaetter:https://github.com/ethereum/pm/issues/1787
00:05:59
Karim T.:Gas used or number of read per transaction ?
00:06:08
Jared Wasinger:Gas used
00:06:57
Karim T.:Asking if it will be better to track the amount of read instead . Maybe not
00:08:28
Karim T.:Maybe we need more tests and push that in another fork
00:13:50
mirgee:Doesn't optimizing execution result in practical benefit only if state root computation isn't the bottleneck anymore?
00:13:56
raxhvl:If we wanted we could infer computational effort by just size of state access
00:15:20
Toni Wahrstaetter:Replying to "If we wanted we coul..."
yeah this works better than gas limit. I analyzed that once
00:15:57
draganrakita:Is this about fast failing on gas mismatch?
00:18:35
draganrakita:Even gas is not reliable to say that Tx is going to take N amount of time to execute
00:19:44
draganrakita:If tx uses N gas it does not mean it will take M ms, it can revert at the start and take M/100 ms to execute
00:21:06
Gary Rong:but in this case it's a bonus for block execution? it's supposed to use m ms but it ends up with m/100 ms to complete N gas
00:21:19
Gary Rong:We actually only care about the worst case
00:21:22
mirgee:Doesn’t gas account for storage growth as well, not just execution time?
00:22:42
Stefan Starflinger:https://notes.ethereum.org/@ethpandaops/bal-devnet-0
00:22:58
draganrakita:Replying to "We actually only car..."
True, just adding a comment about reliability of used gas
00:23:25
Karim T.:Also last version of BAL tests are failing much more with the old forks
00:24:32
Jared Wasinger:Get also doesn’t pass the old fork tests
00:24:41
Jared Wasinger:*geth
00:26:20
mirgee:I think there are some known issues with the tests which should be corrected in the next patch release?
00:26:33
Karim T.:Because when I see the number of failing tests for each clients I will prefer to have more passing tests before https://hive.ethpandaops.io/#/group/bal
00:26:58
draganrakita:I have list on test that fail on v1.4.0 here:
https://discord.com/channels/595666850260713488/1434985857823146059/1435007236563931166
00:27:37
draganrakita:Would assume all/most of them are test problems
00:28:51
Toni Wahrstaetter:This is the PR re this:
https://github.com/ethereum/execution-specs/commit/79ce5c4a49fd86a35204695487cd1c7cecfc4f03
00:29:54
felipe:This is the gas charge split https://github.com/ethereum/execution-specs/pull/1751
00:30:12
raxhvl:Mark we created a thread on R&D to help Erigon with testing, feel free to post any questions related to testing.
https://discord.com/channels/595666850260713488/1435636875321933885
00:32:03
Stefan Starflinger:For testing can clients please run the evm-fuzz scenario of the spamoor in their kurtosis testnets
00:32:26
draganrakita:I feel that revm is in good place regarding BAL. So if people want to run blockchain tests, they can install rust, clone revm repo (use rakita/bal branch), and run cargo run -p revme — btest path_to_test . On BAL missmatch it should print out full BAL
00:33:19
Gary Rong:https://hackmd.io/@bFEBbZiVSAO0IURh9qzEFg/S1Y-mjBkWe
00:36:32
felipe:Printing the full bal is ideal thank you for that. How is this branch different from `bal-devnet-0`, where we're aligning that branch name for all clients?
00:37:28
felipe:Replying to "I feel that revm i..."
ah this is revm, got it
00:39:24
Stefan Starflinger:Replying to "https://notes.ethere…"
Besu also has a debug endpoint to get a disagreeing block and the generated bal by besu. Useful for comparing. Added it to the above doc
00:39:40
draganrakita:Replying to "I feel that revm is ..."
Here is example of output: https://gist.github.com/rakita/e1e2d0dd26d9ab878235ff2be5defa94
00:39:52
draganrakita:Claude is amazing for this kind of work 😄
00:40:00
Po:What's the random read throughput limit on this SSD?
00:45:21
Gary Rong:For batch read over the raw SSD, the qps is 570218.58
00:48:06
Po:okay, seems the ssd's bandwidth is 2G/s. Say it we test it with FIO in 4k read size
00:48:40
draganrakita:While we are talking about number of cores, we should have minimum spec number for it. Something like, recommending 4 cores, similar how ram and disk are recommended.
00:49:07
Toni Wahrstaetter:https://eips.ethereum.org/EIPS/eip-7870
00:49:38
draganrakita:Makes sense
00:52:32
Marc:maybe will complicate testing if we delay decision too late
00:56:45
felipe:I wanted to bring up debugging in this session. It would be very useful if we had the full bal in debug logs from all clients when there is a mismatch. In some cases we are only getting the hash mismatch which makes it difficult to track / compare test BALs vs the client BAL to see where the issue may be.
01:01:45
Jared Wasinger:Geth has a debug method for getting a JSON-encoded BAL via block number or hash
01:01:52
Jared Wasinger:debug_getBlockAccessList
Summary
20 highlights
· 3 action itemsExperimental
Summary
20 highlights · 3 action itemsExperimentalspecification changes
testing progress
performance optimization
- Gas used list proposed for BAL to optimize transaction scheduling00:04:25
- Worst-case: 60M gas transaction delays ~50% vs perfect scheduling00:08:05
- Dragan: 50% speedup difference not worth added complexity00:09:17
- State reads debate: Gary's benchmarks show 8-10x improvement with BAL reads00:33:19
- Read performance plateaus at 32 threads; 10x gain vs 4 threads00:39:09
- Keep reads in BAL for now; revisit with larger state benchmarks00:48:06
client updates
- Geth: Ready except coinbase tracking issue remains unresolved00:23:05
- Besu: Passing Amsterdam tests; old fork failures likely test config00:24:09
- Nethermind: Going through new test release; checking failures00:25:06
- Reth: No pending items; ready for DevNet after test validation00:25:23
- Erigon: Implementation testing; many Hive fails need investigation00:25:46
debugging infrastructure
organizational
- In-person BAL session during DevConnect; date TBD with remote access00:58:27