Maria Silva:So, I think we can maybe wait one more minute to see if more people join, and if not, we'll start.
Transcript
Maria Silva:And in the meantime, I will share… The agenda…
Maria Silva:For the new people joining, just to warn that we are just waiting a bit to see if more people join, and then we'll kick off.
Maria Silva:So I don't think more people are joining, so I think we can just, kick off.
Maria Silva:So, the first, topic is 7904.
Maria Silva:So this is still very, preliminary, so it's not the final numbers, but I just wanted to show you, the recent runs we've been doing. So I'll just…
Maria Silva:copy the link here. So this is the… I put in the link for… for the meeting, but, this is the overview of
Maria Silva:how the numbers look after the Osaka fork. So the numbers we currently have in the EIP were still being run
Maria Silva:On Prague, and so they didn't have the transaction limit EEIP.
Maria Silva:And so we would expect numbers to be a bit better, after this fork.
Maria Silva:And, in fact, we can see, for instance, on the pre-compiles that now a lot of them show, price decreases.
Maria Silva:Another… but still, this is something we need to double-check.
Maria Silva:But, so the main changes between these numbers and the previous ones, or the current ones in the EIP, are, yes, the Osaka fork. Second, Besu did implement some
Maria Silva:optimizations on, mods and div, related operations, and so we should, we should be seeing that here as well.
Maria Silva:We also did some fixes to some of the mods tests, because they were, so the inputs they were using were sometimes leading to some degenerations that actually
Maria Silva:Students…
Maria Silva:they were not, like, really estimating the full cost of the operation, and so now we kind of fixed that. And so the… you can see one big change is that I think numbers between the various mods are not… are a bit more consistent than in the original proposal.
Maria Silva:And also, another, change was…
Maria Silva:So we have, in the tests, we do these loops where we run the same opcode, the target opcode, many times. So, for instance, if it's a mod, we are running this mod many times.
Maria Silva:But because of how we need to set up the test, we need to provide inputs to the operations, so on and so forth, we many times have these self-scaffolding opcodes that get bundled in the runtime when we are estimating the individual runtime of a single operation. And so now we are also
Maria Silva:Excluding, excluding these kind of scaffolding, opcodes from… from the estimation, so we get a more accurate.
Maria Silva:value on the actual runtime of that target opcode. And so this was something that we included as well, and we… we still… so…
Maria Silva:For the Osaka fork, we are still missing data from Erigon and Reth.
Maria Silva:So the tool that we are using to do this is the one that the Nethermind team has been developing, and I think there was some issues running these most recent tests on these two clients. I don't have a lot of details on this, but maybe something to check after if you are curious.
Maria Silva:Also, there were some caching issues on some precompiles, so especially for EC Recover and EC pairing, we know that because the test is set up in a way that is calling the same opcode with the same inputs multiple times, there is likely some caching happening, and actually.
Maria Silva:The clients are not recalculating things, and so this will lead to
Maria Silva:Numbers that look better than they are in reality.
Maria Silva:And then there's also, to have the final, final numbers, we still need to measure the performance on ball-optimized, implementations.
Maria Silva:And these will likely Rethuce the cost increases as well, because we expect, the parallelization to, up here. But just to give an update, so again, this is not the final numbers on, 7904.
Maria Silva:Oh, and I realized that I was not sharing my screen, and I meant to do it, sorry. I'll share it now. So these are the… these are the… the… the latest numbers for 7904.
Maria Silva:And if you see something that jumps to you, it'll be great to have some feedback async.
Maria Silva:Or if you already have some, some feedback, also to, to also have it in the call.
Maria Silva:But yeah, are any comments here, or should we move to the next section?
Ameziane Hamlat:So, I already shaReth in, in the chat,
Ameziane Hamlat:just quickly on keychak, so it is cheaper, right? So the new cash would be 20 instead of 30. Right. And this was one of the opcodes that is actually pretty slow currently, so it was a bit surprised to have it cheaper.
Maria Silva:Yeah, me too. I think this is one of the cases that I think we should look into the test more… more carefully. From a first analysis I did, I don't think there's any obvious
Maria Silva:Hashing happening, because the inputs are always different.
Maria Silva:But this is one that I definitely think we should double check and see if there could be any weird behavior from clients on these tests. I'm happy to share in the report, I can share the link after. I have the actual test that was being used to estimate this.
Maria Silva:And so we, we, we can probably, yeah, if, if you, if any of you have time to look into it and… and give your thoughts, it would be great. Yes, Luis?
Luis Pinto | Besu:Just one remark on the Div and S, Div. Besu has a PR app for adding a new implementation based on amounts, optimizations.
Luis Pinto | Besu:Which hasn't gone in yet, so…
Luis Pinto | Besu:I would expect it to equalize more with not, because the algorithm is basically the same.
Luis Pinto | Besu:Young.
Luis Pinto | Besu:So maybe back to finding people license.
Luis Pinto | Besu:Speaking all.
Maria Silva:Yep.
Maria Silva:Yeah, makes sense. I think, again, we still need to also do the final run with the ball-optimized operation, so for sure, these are not final numbers, so we still have time to take into account all the optimizations you are still working on.
Maria Silva:Perfect. Any more comments here, or should we move to the next section?
Dragan Rakita:I have a question, on what client was this run?
Maria Silva:on all of them, except Erigon and Reth, that were having, out-of-memory issues, if I recall correctly.
Dragan Rakita:So…
Maria Silva:Yeah.
Maria Silva:Gabe?
Maria Silva:Besu… And Nethermind are the clients that were run here.
Dragan Rakita:okay, maybe we can, like, talk in async?
Dragan Rakita:About why… I was, like, out of memory.
Dragan Rakita:But yeah, okay.
Maria Silva:Yeah, I think probably we could, I… so, Rafael from IthpandaOps and Camille from NetherMind, they are the ones currently running the tools to run these benchmarks, so they would be the right people to reach out to, but I think… I'm not sure if they were already in contact with your team or not.
Dragan Rakita:Yeah, sure, but this is on the state load network.
Maria Silva:Yeah, yeah, yeah. Okay.
Dragan Rakita:Oh, okay.
Maria Silva:Perfect. Just comment. It would be really nice if this can be reproducible.
Dragan Rakita:Because this doesn't depend on the state or anything like that.
Dragan Rakita:Maybe having something like the state test or something like that, that can be run and just, like, checked.
Dragan Rakita:A diss runs and times slower, faster, whatever it is.
Dragan Rakita:So you have… every client, like, has a slightly different implementation, and it depends on… You know, like.
Dragan Rakita:implementation depends how fast or slow it is. So, we need to be aware of that.
Maria Silva:Right, so I think… so my part of the analysis, the code is open source, I'm happy to share it after, so you could run these
Maria Silva:By yourself. I think for the data collection.
Maria Silva:I think it's a bit more complicated, but my understanding is that both NetherMind's Gas Benchmark Tool and Benchmark Core
Maria Silva:which is the one being developed by Fail from Itpendops. They are also open source, so you could also set them up and run it yourself.
Maria Silva:I'm… I'm not sure if someone in the call has more information on these that can chime in, but I… all of this should be reproducible.
jochem-brouwer:Yes, maybe also to mention, that these, tests, they are the compute tests, so they are run on empty state.
jochem-brouwer:And I think…
jochem-brouwer:that you can… yeah, you can also run these benchmarks yourself, so, and, like, on Benchmark or on GasBenchmark to do, like, these local tests. These tests are also reported on both
jochem-brouwer:Benchmarker and on, via the guest benchmarks, so you can also see, the results of different runs, except
jochem-brouwer:Yeah, okay, we… I don't think they are targeted at a performance bench currently, so that's something we should maybe check, yeah.
Maria Silva:I actually… I think for Benchmark Core, at least, I think they are. I remember seeing that they are tracking better performance bench.
Maria Silva:And, just to clarify, actually, for these numbers, they were run on top of maintenance states, just because we were, at the same time, also collecting some
Maria Silva:state benchmarks that needed at, but it shouldn't make a huge difference, because the… these compute ones shouldn't have, like, a huge…
Maria Silva:Like, having it run on my net wouldn't make a lot of difference.
Maria Silva:Perfect. Any more comments here?
Maria Silva:So, Lucas, to answer your question, this doesn't take into account wall-optimized branches.
Maria Silva:I'm assuming there's still temporalization going on, but I think the main difference was actually the Osaka fork and the 60 million.
Maria Silva:Gas limit transaction.
Maria Silva:And I think there might still be some…
Maria Silva:Issues with some of the pre-compiled tests, and we should look into them as well.
Łukasz Rozmej:Okay, so…
Łukasz Rozmej:Oh, those numbers actually puzzle me, because I expect that we will be increasing a lot of precompiles, and here we are kind of decreasing most of them.
Maria Silva:Yeah, yeah, yeah. So I think at least I know for EC pairing and EC Recover, these numbers are bogus, because essentially we are doing the same
Maria Silva:inputs, so there's definitely caching going on. For the other ones, I think we need to double-check and see if the issue is in the test definition, or how they are being run.
Maria Silva:But yeah, we need to… to… to double check on that.
DanielVF:Yeah, as I said in the comments, Monte had instrumented these and found that, these EC methods were much more expensive.
DanielVF:And drastically raise the prices on them.
Maria Silva:Yep.
Łukasz Rozmej:Yeah, so this… again, this is completely not what we got from our benchmarking some time ago, from our benchmarks. I don't… I'm not sure what the current state is, Camille is…
Łukasz Rozmej:Leading that, is he here? Why isn't he here?
Maria Silva:I don't think it's…
Łukasz Rozmej:Maybe I should ping him. But that's, like I said, completely the other way, that's one thing. And second thing, I think we shouldn't, for example, decrease pre-compile costs now if we, for example, will be increasing them very soon due to ZZK cost, because that's kind of…
Łukasz Rozmej:Doesn't make any sense.
Maria Silva:So, yeah, just a quick comment on that. I completely agree. So, anything that is here with a cost decrease, if there actually were, at the end, with the final numbers and everything, a cost decrease, the proposal would be not to change it. So this is just the preliminary numbers for you to get a sense, and I think we'll likely
Maria Silva:Find some issues with these tests, or how they are being run, that will…
Maria Silva:Tell us why these numbers are having a decrease rather than increase.
Łukasz Rozmej:Okay, okay, that's fine, that's fine, but again, the two are completely… numbers don't agree with…
Łukasz Rozmej:my previous knowledge. Right.
Maria Silva:Yeah, agreed, yeah. Joram, do you want to go next?
jochem-brouwer:Yeah, so Daniel and Lukas, you said that,
jochem-brouwer:yeah, your benchmarks say otherwise. I also think that These numbers shall be,
jochem-brouwer:well, at least be increased, but could you maybe share these benchmarks, so we can also check? And I think, like, this, this caching thing, it's interesting because this cache thing keeps, hitting us, because we don't want the best case here, we want the worst case, and this is, well, very hard to,
jochem-brouwer:to achieve using, like, the EVM level benchmark. So you can't directly go into the client, you have to make EVM tests, and then this cache keeps…
jochem-brouwer:hitting us, but yeah, okay. If you could share, like, the, these benchmarks, that would be great.
Łukasz Rozmej:Yeah, those were the, you know, the old benchmarks… well, I don't remember, again, I pinged Camille to… if he can… if he can join. I don't remember, I don't know the current state, I don't follow. I remember that we had those benchmarks when we created the block.
Łukasz Rozmej:And we pushed, you know, the block with,
Łukasz Rozmej:I probably are aware of it, right? That those are the earliest benchmarks that we used in Berlin on the Interop.
Maria Silva:And we pushed the block full of.
Łukasz Rozmej:opcode. Of course, you know, it needed, for example, also opcode to manipulate the stack, because those were kind… kind of real blocks. So it wasn't just isolated opcode, but the opcode sequence that allowed us to use this opcode. And we used,
Łukasz Rozmej:Data that isn't repeating, so it can… couldn't be cached, right?
Łukasz Rozmej:So, each opcode should have, each user.opcode should have a different data, right? So, slightly changed, so the cache couldn't be used. Is that how it worked? I don't know… I know those,
Łukasz Rozmej:like, those tests were trying to migrate it to eels, etc. I have no idea what the current state. I know that, this migration, I heard that there were problems, for example, with, with this, non-repitability, this, this, caching…
Łukasz Rozmej:prevention strategy, I know that there was a problem. I have no idea what the current state, Camel, Camel knows that. I checked out a few months ago on exact details.
Maria Silva:Right, and I…
Maria Silva:I think on that, I definitely know that for, again, EC Perry and EC Recover, they are using the same inputs in the current tests.
Maria Silva:So that's definitely something we need to change. But I would also say that right now, with Osaka, you can't fill a block with just one transaction, right? So you would at least get, like, around, like, 4 transactions, right? So you need to split it by the 60. So you would already expect to see
Maria Silva:Some performance gain there, because now you can… That was my…
Łukasz Rozmej:Question, do you… those… does those tests, take into account parallelization?
Maria Silva:That was my question, and… I would assume that you would use the default for the client, right? Like, now you have a block with, like, a certain number of transactions that have the 60 million
Maria Silva:Limit, and then it depends on how the client's, place.
Łukasz Rozmej:But, okay, but for example, you can make a block this way that, because of previous transaction, the new transaction will have a slightly different data into that, going into that.
Łukasz Rozmej:Into that, pre-compile, which makes it more or less non-parallelizable, right? Until you get balls, right?
Łukasz Rozmej:With balls, it will make it not probable, so… you can still attack it, right?
Maria Silva:That makes sense, yeah. I don't know how this… what is the status on that, on these tests,
Maria Silva:I don't know if someone in the call knows, but.
jochem-brouwer:Yeah, this, this is… Yeah, this is definitely something to, to, check,
jochem-brouwer:very, soon. But I also realized that
jochem-brouwer:NetherMind has a decent strategy to do, like, the parallel execution, and then there's also, like, a pre-compiled cache, I think, at the transaction level.
jochem-brouwer:And I realized that, for the Osaka test, where you actually spit up your transaction into multiple, this will actually make it perform, like, very good if, like, each transaction does not change their inputs
jochem-brouwer:Depending on the result of the previous transaction, because now you can use
jochem-brouwer:Parallelize all these transactions, cache them, and then when the transaction actually hits in your sequential run, then you can just read it from cache.
Łukasz Rozmej:Yeah, so we actually have a setting, I think, we added to disable this cache, if you want that, because we just wanted to have it for test, and Camille should join in a few minutes, so we can actually ask him more questions about…
jochem-brouwer:If you could post this flag, and then I will check if this is already activated on the benchmark, because this is something we definitely need. But also, I think we should integrate this in the test anyways, because
jochem-brouwer:Yeah, like…
jochem-brouwer:We should generate these worst-case blocks, and also think about, like, client optimizations like this, and to ensure that this kind of caching cannot
jochem-brouwer:happen, and…
jochem-brouwer:I… I'm not… I don't think we have this yet. That should be added, but thanks. If you could post this flag, that would be great.
Maria Silva:Dragon, do you want to go next?
Dragan Rakita:Yeah, I would like to propose to remove changes to precompiles.
Dragan Rakita:precompiles…
Dragan Rakita:There are a lot of nuisances there, depending on the cache, depending on the library that's used.
Dragan Rakita:Depending on the implementation that's used.
Dragan Rakita:It feels… I feel that you…
Dragan Rakita:A lot more time to be sure that they are okay.
Dragan Rakita:So I think removing them, it's… at least I'm proposing it to remove them from this EIP.
Maria Silva:Yeah, I'm not 100% sure we can do that. Like, if you just… if we just…
Maria Silva:Like, blankly remove them, then we may not be able to increase the gas limit, right?
Maria Silva:But…
Dragan Rakita:What do you mean by that?
Maria Silva:I, I mean that if…
Maria Silva:So, for instance, I… again, I know EC pairing and EC Recover, they are being cached, for sure, because the inputs are… are… are… are repeated. So, if they actually require a cost increase, like.
Maria Silva:two times cost increase. This means that they will be bottlenecks that we are not repricing, and so they will be blocking us to raise the cost limit.
Maria Silva:Or were you meaning that if the numbers lead to a gas decrease, then we shouldn't include them? And if that's the case, then I agree. So, as I was saying.
Dragan Rakita:I'm also…
Maria Silva:As I was saying before, this is just the numbers that I'm showing, but the rationale would be, if for some reason, with the final numbers, we've still figuReth out that some precompile was more efficient than we thought, and it would require a gas decrease, then we won't change it. Like, here we are just making gas increases in this EEIP.
Dragan Rakita:Yeah, to be honest, make the worst case be coveReth by gas is the, like, the best case for us.
Dragan Rakita:If we can manage that, that would be great. And yeah, I agree that we need to do that if we want to increase the block gas limit.
Dragan Rakita:So yeah, we should include those.
Dragan Rakita:for example, the Blake…
Dragan Rakita:pre-compile, you have, like, instruction, AVX instruction that you can use to speed it up, or you can use, like, default implementation that's slower.
Dragan Rakita:It's, like, very… I have both of those inside the REVM, but depending on what platform is used, it could be one case or another case.
Dragan Rakita:So we want to, like, for precompiles, we want to, like, match them in the… for the worst case.
Dragan Rakita:So there are a lot, like, nuisances around that. We should be careful. But yeah, I agree with you, we should definitely increase it, because we need it for the increase on the blog, yes, I mean.
Maria Silva:Right, and I think this is exactly the reason why
Maria Silva:I think it would be helpful to have more of you looking into the tests and see if there are, like, any particularities in your client implementations that might mean that we are not really capturing the worst case.
Maria Silva:Like, any help here at this point, I think would be much appreciated.
Maria Silva:Conscious of time, I think we should move to the next topic, so maybe if there are any more comments, we can continue these.
Maria Silva:async. I think the next update is around 1838, and Joram was going to chat a bit about the benchmarking of state operations, and, the update there. Joram, do you want to go next? I'll stop sharing.
jochem-brouwer:Yes, let me share my screen.
jochem-brouwer:Wait, this… boom, boom.
jochem-brouwer:Interesting.
jochem-brouwer:It's horrible.
jochem-brouwer:Alright, can you see my screen?
Ameziane Hamlat:Yes.
jochem-brouwer:Yes.
jochem-brouwer:Cool, okay, great. So this is about ERP 8038, about the benchmarking state operations. So, a quick recap, it's about the, about the change of the state access operations.
jochem-brouwer:These are the parameters which it is targeting, so this is anything which has to do with changing things to the state, so this has to do with contact, storage.
jochem-brouwer:And accounts.
jochem-brouwer:And the nasty thing here is…
jochem-brouwer:again, the cache. So we just talked about the compute, caches, which are, well, they can be nasty, or, like, they, they, they optimize the, the benchmarks.
jochem-brouwer:And that's not what we want, because we are actually targeting the worst cases here, so we have to take care of these caches. And in the EVM, we have the cache accounting, so we have cold storage and barn storage, both for the accounts and the storage.
jochem-brouwer:And this is what I wanted to mainly go into, because this is what we have been tackling in the last, well, week, two weeks.
jochem-brouwer:And the EVM cache, this is from ERP2929,
jochem-brouwer:And what it does is it tracks which accounts and which storage has been accessed, and if it has already been accessed.
jochem-brouwer:then… The cost to change this value is, decreased.
jochem-brouwer:So we get, like, a discount.
jochem-brouwer:And this happens on the transaction level. So there's the EVM cache.
jochem-brouwer:For the benchmarks, we had the initial setup.
jochem-brouwer:That, if we want to target existing accounts or storage.
jochem-brouwer:We have this setup where we
jochem-brouwer:Create these accounts, or these, the storage.
jochem-brouwer:Before running the benchmark.
jochem-brouwer:And the reason for this is that in the accounts, but also in the stores, you have to pay extra if you start creating accounts, so this is one of the reasons from the EVM. But also, from the clients, we see that in some clients, reading
jochem-brouwer:The storage or the accounts, which do not exist is actually the worst case for them.
jochem-brouwer:So we need to have tests for both accounts or storage for which does not exist, and for which do exist, for both for the EVM constant or the non-EVM.
jochem-brouwer:content, so like the, the, like, how the database operates, like that, and, also about the caching strategy of this client.
jochem-brouwer:So, what we have seen, and I will call it a cash monster, because we have just seen that this is not nice in the, in the computer test, but it is also very not nice,
jochem-brouwer:Not to… yeah. On the state level. So normally, if we have, like, cache hits on the mainnet, we are, oh, we are, we are, like, it's, like, very nice, because this increases our performance, but if we hit it on the benchmark, this is not very nice.
jochem-brouwer:So, Maria, made these graphs, and you can see on the left the cache hits and the misses without preset keys.
jochem-brouwer:And on the right, you can see it with the preset keys. So, with the preset keys, we actually started to create these keys in, like, blocks right before the
jochem-brouwer:Benchmarks actually being run.
jochem-brouwer:And what we see is, like, the difference between these two is, like, not so nice. So there's a problem.
jochem-brouwer:With, like, this cache, because it looks like, well, most values are actually in this cache, which is not what we want, because we want to,
jochem-brouwer:We want to… to get, like, the worst case, analysis.
jochem-brouwer:Yeah, and just to recap it, I think most of you know, like, the tests I run on top of a mainnet or a BloatNet snapshot, and BlowNet is the network which is a fork of mainnet, and we just spam, like, a lot of state on top of this.
jochem-brouwer:So, what we have to do is we have to battle this cash monster, and the new strategy is to ensure that the tests are actually… they are forced to go to disk.
jochem-brouwer:And this means that the snapshot which we use to run this benchmark on have these keys, so these storage keys and these accounts keys already preset.
jochem-brouwer:And,
jochem-brouwer:Oh yeah, and we have to skip in the test itself, so we don't want to have the set of this in the test that, like, in the previous blocks, before the benchmark, we, the benchmark block, that we actually set up these keys or these accounts, because this already puts them into the cache, and that's not what we want.
jochem-brouwer:So what did you do on bloatnets? And I want to know that this bloating is, like.
jochem-brouwer:Well, very, slow, like, it takes a long time to create, like, a reasonable amount of accounts, or, like, mainly storage, that's, that's, that's the main thing.
jochem-brouwer:So what we do is we build ESC20-like contracts with storage, and we also upload a range of accounts to query EOAs, so accounts without code.
jochem-brouwer:And we also want to target contracts. The reason for this is that we want to target also ERP2780. So 2780 wants to reprice,
jochem-brouwer:intrinsic gas costs, and also gas costs if you query accounts with code or without code. Like, now there is a different… this introduces different costs if you
jochem-brouwer:target accounts.
jochem-brouwer:With code or without code, so we want to directly integrate this.
jochem-brouwer:And, well, as a nice thing, like, for the target contracts, we do not need to upload this, because we can use an old contract on mainnet, which has created many accounts in the past, like, it's almost 430,000 contracts, which is the old ENS PTS3, so that's, like, very nice to resurrect one of these,
jochem-brouwer:old contracts, so we don't have to bloat on BloatNet, like these extra accounts. And the reason that we target the create opcode is because this generates a pattern of contracts, sorry, of addresses into the EVM, which we can directly,
jochem-brouwer:generate in the EVM without, like, a lot of cost, because otherwise we have to hard code the addresses into the EVM, and this is, like, both complex and,
jochem-brouwer:We also have to figure out, then, a way to actually hard-code these things, because we have the contract size limit and all kinds of challenges. So, on BloatNet, we have the… these new steps of different sizes. There's one missing that's already there. There's also 30GB of addresses, and we also want to see, like, okay, what
jochem-brouwer:Does this contact size of the source itself have an impact on the benchmark?
jochem-brouwer:For the benchmark targets, we have 3 levels of cache, and to also see, like, if there's, like, a difference. We have the no cache, so this is the part where we directly read from the snapshot state, so from disk.
jochem-brouwer:the caching at the THX level, that's where we target the state where it is both warm in the EVM and it is warm in the cache, so we actually target the storage slots, or the accounts, into the EVM where we first query these accounts before actually doing the opcodes which we are targeting for this benchmark.
jochem-brouwer:And there's also the caching strategy at the previous block. What we then do is, like, in the previous block, we query these accounts on the storage, and then in the next block, we do the actual benchmarks.
jochem-brouwer:For the accounts, we target EOAs and, contracts.
jochem-brouwer:And we also call these with value and without value, to also see the impact of changing our counts here.
jochem-brouwer:And for the storage bytes, we target these ESC20 targets, the existing and the non-existing keys, because, well, reading non-existent keys in some clients is the worst case.
jochem-brouwer:And also, reading keys is always necessary for storage writes because of this gas cost calculations. So if you create, like, new keys, you have to know if the key already exists or not, and a key which has zero in it, so it's non-existent, changing it to a value is actually expensive.
jochem-brouwer:And, we also check, like, okay, what if we write exactly the same value, or what if we change the value, what is the impact?
jochem-brouwer:For the storage reads, we target different sizes EC20 contracts, so this list which I showed before, and also, like, the existing and the non-existent keys.
jochem-brouwer:So, what are the next steps? So, right now, the broad net step is being taken.
jochem-brouwer:What we then do is we run these benchmarks on top of the snapshot.
jochem-brouwer:And then we analyze these benchmarks, well, if there is a problem with the test, we fix it, and, well, the main question is, and I likely… I would be very surprised if this, result would be a no, but did we fight this cache monster?
jochem-brouwer:Correctly.
jochem-brouwer:Alright, that's my, those are my points. Any questions, points, remarks?
Maria Silva:You can go, Daniel.
DanielVF:Yes, do you have a rough idea at this point?
DanielVF:how much increase you're thinking? I know the EEIP says TBD.
DanielVF:On, like, read access.
Maria Silva:I say it's still TBD, and it's hard to say, because on one hand, we do expect a pri… like, based on previous, benchmarks and how Zen
Maria Silva:We do expect there to… the fact that we need to increase the price, but at the same time, with ball-optimized state reads and, state route calculations, it's hard to say how much more performant clients will get.
Maria Silva:So I think, in my mind now, it's very hard to say what will be the numbers. I would expect
Maria Silva:An increase, but how big it is, it will depend on the ball optimizations, And how…
Maria Silva:Clients will behave after that.
DanielVF:Okay.
DanielVF:Yeah, coming from, Monad and, doing a lot of performance testing there.
DanielVF:State, you know, as you get into thousands of transactions a second, or 10,000 transactions a second,
DanielVF:just the speed that things can be read from the disk very much becomes a bottleneck, and is probably the primary bottleneck, at that… as you get up toward that speed. So, we increased the cost on cold reads, just because it is very much a bottleneck.
DanielVF:But, again, if you're running slower transactions, that may or may not
DanielVF:be the case, or it may not matter quite as much.
Maria Silva:Right, I think it will be great. Once we have the… more numbers on these, it will be great to compare and see how things differ, but I think at this point, it's really hard, because we are still… I think the numbers we have right now are not reliable enough for us to have a good view on it.
DanielVF:Yeah, that makes sense.
DanielVF:I guess just the one thing I want to throw out there is that,
DanielVF:Disk read speed can be another access of, you know, the hard balancing act of getting,
DanielVF:all the different dimensions of gas pricing together, but it's definitely a fundamental Fundamental constraint.
DanielVF:performance.
Maria Silva:Right, thank you. Amisienne, do you want to go next?
Ameziane Hamlat:Yeah, thanks. So, yeah, thanks, Yukam, for the presentation. I think it makes a lot of sense to have, like, these different scenarios. My question is related on how will our product
Ameziane Hamlat:sort of productable are these tests? So I guess the idea is to take, like, the snapshot of the blood network.
Ameziane Hamlat:then run on top of it, like, all these tests, and then we can delete, you know, like, everything that was generated, and then run again, run again. Is that, like, the idea? And the other question is, like, you said bloating is very slow.
Ameziane Hamlat:And I was wondering if it is related to the network itself, or to the, like, the benchmarks. Are we benchmarking the state, like, state creation in a way that we are creating, like, a lot of, basically, state as well?
jochem-brouwer:Okay, so for the, for the first question is, like, how, how, how reproducible are these, these benchmarks? So, these are East, benchmark tests. They are specifically targeted for BloatNet.
jochem-brouwer:But,
jochem-brouwer:if you, like, create, like, an EC20 contract with the functionality which the benchmark needs, then you can also point these benchmarks to these contracts on any network. So you can also generate, like, a pre-state yourself, or a set state yourself, from, like, empty state.
jochem-brouwer:And run it on top of that.
jochem-brouwer:maybe… yeah, yeah, to answer your question, like, how are these benchmarks being run? Like, does it, like, delete all the states when this benchmark happens? Yes, that's exactly the case. So, how these benchmarks are being ran?
jochem-brouwer:Camille could explain better, but I think I… well, if I explain it like this, that this would be,
jochem-brouwer:Correct? Like,
jochem-brouwer:these are, like, clients, you start them up, you run this benchmark, and then you kill them, and then you do, like, the next benchmark. So this should get rid of any caching or, yeah, whatever.
jochem-brouwer:I think that answers your first question. The second question was…
jochem-brouwer:No, I don't. Now I don't remember. But…
Ameziane Hamlat:was related to blue tin. You said gluten was slow, and I was wondering…
jochem-brouwer:Yes. So, the problem with bloating is, like, it's not really about the network or, like, if you want to, like, stress test the network there. The problem is that we are really being constrained there by the EVM. So, the bloating speed's essentially, like, you have to go… you have to do, like, a lot of overhead operations all the time.
jochem-brouwer:So you have to run the transaction, you have to win the blocks, but that also means you have to do, like, the state rules calculation and these kind of things. So this is, like, very slow. If you have to upload, like, 50 gigabytes of state, well, you can already calculate it yourself.
jochem-brouwer:So for 50GB of state, you would need 50GB divided by 32 storage slots, and creating storage slots is 20,000 GAS.
jochem-brouwer:So you can calculate how many guests you would need to actually build this thing, and that's, like.
jochem-brouwer:the hard low limit of this, and of course you have, like, some extra overhead, like, intrinsic transaction cost, the Osaka thing where you have to split the transactions. Well, you can imagine that this is not very fast to do this.
jochem-brouwer:Okay, thank you.
Maria Silva:Thank you. So, conscious of time, I think we'll move to the next, section, and if you still have any comments or questions, please let's do it a sync. I think now we have Spencer. He'll, talk a little bit about 8037, so the State Creation Gas Costing for CEIP.
Maria Silva:And his update on the specs and tests for Definite 3, and then also we'll chat a bit about
Maria Silva:The…
Maria Silva:changes they are doing to the entire testing framework to then accumulate all the repricings for DevNet 4. Spencer, you want to go?
spencer-tb:Hey, thanks. Thanks, Maria. Yeah, so… last…
spencer-tb:last week on… I think it was Saturday, we…
spencer-tb:released, or DevNet Free release, with…
spencer-tb:Our, like, initial set of tests for… 807, and
spencer-tb:Yeah, it seems… seems to… seems to have gone well, like, I'm glad, clients are passing, I'm glad Besu's passing most of those tests, which makes me happy.
spencer-tb:With this release, we have…
spencer-tb:Hard-coded the cost per state byte to, 1174, which is what it equates to when you use a block gas limit of 100 million.
spencer-tb:And…
spencer-tb:So the reason we did that was because of, complexity in the framework issues, and we wouldn't be able to get a release out, on time before the definite.
spencer-tb:So… For the next EVNET, at least over the coming weeks, we plan to…
spencer-tb:Yeah, update our framework to handle the EEIP correctly.
spencer-tb:So the issue being, that we've never had this,
spencer-tb:I guess, extra dimension of the block gas limit affecting gas costs within our framework, and it affecting a lot of tests, and I'd say the biggest difficulty has been the, static tests. We call them static tests, basically the…
spencer-tb:tests from Ethereum tests, they're YAML files that we are filling to JSON, so we have, like, a separate way of doing that.
spencer-tb:And a lot of those tests were failing.
spencer-tb:The hard part is that, yeah, changing those tests, to…
spencer-tb:past the spec for 8037 is not really possible in YAML, so, we've made good progress on porting these tests to Python, and then once they're in Python, we're going to update the logic for these tests for 8037.
spencer-tb:I'd say the overarching
spencer-tb:goal for me, and, I guess for DevNet4 is to, integrate, 8037,
spencer-tb:78… no, 2780, 8038, and 7904, all into the framework with all the tests passing. Really just…
spencer-tb:Hardening everything so that… Yeah, the framework's more robust.
spencer-tb:And I think we should be in good shape for DevNet 4.
spencer-tb:Yeah, that's, that's my update. I can keep it brief. If there are any questions, please ask. I guess, I also have a question.
spencer-tb:M… Regarding, benchmark tests for 8037,
spencer-tb:Do we want any specific tests to target the…
spencer-tb:Reservoir mechanism, or should we…
spencer-tb:Or are we okay with the existing, State operation…
spencer-tb:tests that we have, for a blood net. I'm not super deep with those, so that's, what I'm asking.
spencer-tb:Thanks.
Maria Silva:Yeah, I think that's a great question. So, the tests on BloodNet are really just…
Maria Silva:benchmarks, they are not really tests, so they are just trying to assess the execution time of all state operations, so I don't think they are good to
Maria Silva:test that the clients are implementing 8037 the way they are supposed to. And so for that, I do feel like we need some sort of tests to
Maria Silva:to…
Maria Silva:to test. I don't have a better word for it, but we need tests that, kind of allows us to see these new code paths, using the reservoir.
Maria Silva:So I think it would be… it would make sense for us to have something around that.
spencer-tb:Sure, sure. Just to clarify, so we have… we have tests to make sure that clients are up-to-date with the spec, but for…
spencer-tb:benchmarking the reservoir specifically, we don't, so if… if we… if we need some of those, I can, take a note and take a look at that as well.
Maria Silva:So then, can you… can you explain what you mean by benchmarking the reservoir?
Maria Silva:What do you think is missing from the tests you already implemented?
spencer-tb:Well, the… the tests so far are just, like, spec compliance tests, so there's no…
spencer-tb:Like, I assume… I'm assuming the existing tests that benchmark,
spencer-tb:I guess what I'm trying to say is, because there's this kind of new code path.
spencer-tb:With the, the second dimension of gas.
spencer-tb:Should we have an extra… should we benchmark that specifically?
spencer-tb:Or are the existing tests that we have enough?
spencer-tb:To cover that.
spencer-tb:My understanding is it's the latter.
Maria Silva:Right, I think my understanding is also the latter, so I would assume that
Maria Silva:Adding this second, dimension, would it increase
Maria Silva:significantly execution time of things, but if… if it does increase, you'd still see it in the test that we are running right now, because we are considering the variance where
Maria Silva:the, for instance, SSTOR creates new slots, and so we would be able to observe on that test, like, an increase in
Maria Silva:in execution time. So, definitely once… and it's something we can definitely do, like, once we have a DevNet with all the repricings.
Maria Silva:can Retho all the benchmarks and see if there are any new bottlenecks, but I wouldn't expect
Maria Silva:us to need… new tests,
Maria Silva:Because with the tests we already have, we should be able to… to see it.
Maria Silva:yeah.
Maria Silva:I don't know, Jocham, you're going to say something as well, right?
jochem-brouwer:Yeah, yeah, yeah, yeah, yeah. So, so, okay, so I'm not… Deep into, 8037.
jochem-brouwer:like the current spec. But what we should do is we should… so we have, like, the spec test, and we have the benchmarks, and for the benchmarks, what we target there is, like, we want to target the worst case.
jochem-brouwer:What we should check.
jochem-brouwer:is if this ERP introduces, well, new code paths, or, like, new ways to, abuse, export, or, like, are there, like, new scenarios
jochem-brouwer:Where we have, like, these worst cases happening.
jochem-brouwer:And if there are new scenarios which are then just not coveReth, or likely not coveReth by the benchmark tests, or the benchmarks, then these should be added.
jochem-brouwer:So there should be, like, an analysis of this.
jochem-brouwer:The question is.
jochem-brouwer:who is going to do this? Because I don't think I have the bandwidth to do that right now.
Maria Silva:Right, and I, I do think there's… first, on, on, on, on, on one sense, there is…
Maria Silva:The question of, now, state creation gas is not included in the transaction limit of $60 million.
Maria Silva:Gas units, so we could actually have
Maria Silva:New worst cases that we should think about.
Maria Silva:So yeah, maybe it's something to put as a to-do, and then try to dedicate some time to think about this.
spencer-tb:I can, I can take a look over the next week.
jochem-brouwer:Cool, cool.
Maria Silva:Thank you, Spencer.
Maria Silva:Great. Any more comments, or can we move to the next section and lasts?
DanielVF:Yeah, one question I have, given that
DanielVF:Block times are scheduled to… are desiReth to go down, and probably will go down.
DanielVF:How does this interact with that? Because it seems like if you just follow the formula, we're actually making state, cheaper as block times go down.
Maria Silva:Yeah, so it would adjust with the… With a block limit.
Maria Silva:I think once we do shorter slots, we would have to refactor that in. Because at the end of the day, what we want is to keep a steady… like, we want to enforce, like, a long-term state growth.
Maria Silva:So, now if we are doing…
Maria Silva:more… So that the… the sort of the… the block space should, like, the… the cost in relation to the aggregate block space should become constant, shouldn't… shouldn't change. So once we are doing the zigzag or the short… if we end up doing the short,
Maria Silva:Slots, we need to… Do the analysis and likely update the cost per byte.
Maria Silva:But I think we'll… we'll… we'll hit that when… we will hit that… we'll cross that… that bridge when… when we get there.
DanielVF:Green.
Maria Silva:Perfect. So, conscious of time, any more comments, I'll,
Maria Silva:ask them to be async, and I'll now call Bhuta. He has an important update on the community outreach for repricings. Bhuta, the floor is yours.
Butta:Yes, thank you. Now, I feel bad for not having a nice presentation, just like you have.
Butta:But, yeah, for the survey itself, right now we have, 19 different entities who filled it out, mostly application developers, around 70% of them are application developers.
Butta:Other ones are, yeah, wallets, validator operators, or node operators in general. In summary, I would say, so for… I will, I will go through each EEIP, so it's a bit more, clearer.
Butta:But for the first 5 EEIPs, which is 79.81,
Butta:7976, 7904, and 7778. There were no controversial opinions, so most of them, if not all of them, were fine with it.
Butta:With 2780, there was one question, I would say, which also led to a significant remark from them, asking whether
Butta:Deploying contract is also affected by it, because if it is, then also smart wallets will be affected, so maybe that's a question.
Butta:From Maria. Something we also did is, or I did is, everyone who had a high concern with any of the EEIPs, I discussed it with Maria and then reached out to them to clarify.
Butta:How they're affected, and if they're affected at all.
Butta:So, maybe this is something we can discuss afterwards, Maria. For the last two EEIPs, 8037 and 8038,
Butta:these were more the controversial ones, because, I think… so out of the…
Butta:19 people, 6 were very concerned that it's extremely complex, to integrate, and also to use. Maybe also worth noting that, the survey was filled out by larger entities like Lido, Yearn, Arvware,
Butta:handle, like, more of the DeFi applications. So, yeah, I think 8037 is the most controversial one.
Butta:But yeah, we reached out to them, so depending on whether this EEIP actually makes it in, this is the one that, yeah, I would say is a bit questionable.
Butta:Right now.
Butta:As for the next step, so this is me reaching out in general, we will see which of the entities are actually affected. So some of the data was already deliveReth by Tony, where we reached out to some of the Layer 2s. I think it was,
Butta:One was Polygon, and one I would… I'm not allowed to mention, I think. I also reached out to Arbitrum and Optimism, and see what they will…
Butta:think about, generally, the gas repricing. So yeah, the Layer 2s will also be involved in… in their opinions, and the big…
Butta:the big stakeholders we're missing right now are wallets, which are a bit harder to reach out, it seems, but I will try my best to get their opinion during EFCC.
Butta:And then we will have the big picture. And in the final step, this will happen…
Butta:End of this month, ideally.
Butta:is when Carl provides his in-depth data, and to see who's actually affected, we will be able to reach out to affected entities specifically.
Butta:Yeah, I think that summarizes. Ideally, next time I have also a nice presentation.
Maria Silva:Yeah, I would just add, and thank you, Bhuta. So, my understanding was that the two biggest concerns around the state's operation EEIP, so 8037 and AT38,
Maria Silva:Was, on one hand, the actual increase, so a worry that making things more expensive could…
Maria Silva:Hurt some applications, either because things get too expensive, or because there's some, like, cold pots that need to be updated.
Maria Silva:And the second one was on the cost per byte, so the dynamic… the fact that we have a dynamic cost per byte now leads to some more complexity when
Maria Silva:Optimizing,
Maria Silva:apps, and so I think on the final point, it's, to me, the one that we should, and I think it's something we'll do in the next steps, as we follow through with these entities, is trying to understand, like.
Maria Silva:With the… so we do have now a quantization trick that makes the…
Maria Silva:dynamic cost per byte a bit smoother, so only changing when there's big, increases, so if this already makes things better, or,
Maria Silva:Or not, and then, actually, how much worse does it get in their overall experience? Because sometimes things, just because they are complex, they seem
Maria Silva:More complicated, but then, like, the final… Day-to-day impact is… is not…
Maria Silva:that's, big, so I think it's, it's, it's understanding, the actual, the actual concern by getting more
Maria Silva:more context,
Maria Silva:But yeah, so those… those… I just wanted to add those two points. And, Tony, do you have, something to… to add?
Toni Wahrstätter:Yeah, just to… just to quickly out, to add, the one entity that Buda left out is Starkware, and what I was meaning with you don't need to reach out to them was more because I already reached out to them, so not, like.
Toni Wahrstätter:they don't want to be mentioned or something. It's very obvious that Starkware, who is not using blobs, is affected there.
Maria Silva:Thank you. Are there any more comments or topics?
Maria Silva:Great, thank you so much, then, for attending, and we'll see each other in two weeks.
jochem-brouwer:Thanks, Harold.
jochem-brouwer:Bye-bye.
Louis:Thank you.
Chat Logs
00:02:34
Louis:gn
00:02:52
Maria Silva:https://github.com/ethereum/pm/issues/1959
00:04:16
Maria Silva:https://github.com/ethereum/pm/issues/1959#issuecomment-3996071514
00:06:38
Ameziane Hamlat:So keccak is cheaper in the new results. It is one of the one opcodes that needed repricing. I was expecting more expensive
00:08:03
Louis:For Reth & Erigon, I think the issue is OOM
00:13:42
DanielVF:Interestingly Monad instrumented and considerably increased the prices on the EC methods
00:15:12
Łukasz Rozmej:those numbers are because of parallelization? Otherwise I expected different ones.
00:19:47
Ameziane Hamlat:Yes, I remember there were benchmarks in gas benchmarks with cachable precompiles and not cachable precompiles
00:24:03
Łukasz Rozmej:--Blocks.CachePrecompilesOnBlockProcessing=false
00:24:24
Ameziane Hamlat:Besu does optimistic parallel transaction execution by default but we can disable it as well.
00:24:59
Kamil Chodoła:Jochem - Nethermind runs on benchamrks with that flag already - Reth probably have this cache disabled also
00:25:10
Łukasz Rozmej:--Blocks.PreWarmStateOnBlockProcessing=false to remove background pre-warming (of state and precompiles too)
00:25:13
jochem-brouwer:Antwoord verzenden naar "Jochem - Nethermin..."
Ah right thanks!
00:27:00
Kamil Chodoła:But not sure if you really want to disable prewarmer?
00:27:06
Ameziane Hamlat:Replying to "Besu does optimistic..."
--bonsai-parallel-tx-processing-enabled=false to disable it
00:30:59
Maria Silva:FYI, plots were taken from benchmarkoor
00:31:06
Maria Silva:It has very nice visualizations
00:34:17
Luis Pinto | Besu:I think regarding vectorisation use in benchmarks, I guess we can assume that most people nowadays have access to at least AVX2 enabled chipsets at least for intel ones. Re: home stakers
00:41:48
Louis:This is the benchmark for BALANCE, EXTCODE*, CALL*
https://github.com/ethereum/execution-specs/blob/3f507fa91fce7ebb8fa00537dfc709961bd582af/tests/benchmark/stateful/bloatnet/test_single_opcode.py#L1543
00:44:07
spencer-tb:https://github.com/ethereum/execution-spec-tests/releases/tag/bal%40v5.2.0
00:46:46
Maria Silva:Great progress, Spencer!
00:57:25
DanielVF:I'm in favor of an increase in storage gas, but am quite concerned about the complexy/benefit on the scaling gas prices.
Summary
12 highlights
· 4 action itemsExperimental
Summary
12 highlights · 4 action itemsExperimentalrepricing updates
- EIP-7904: Post-Osaka numbers show precompile decreases; likely caching artifacts00:04:11
- Proposal: Remove gas decreases from 7904; only increase bottleneck opcodes00:23:58
- EIP-8038: Cache monster battles ongoing; preset keys force disk reads00:29:02
- DevNet 3 released with 8037 at hardcoded 1174 gas/byte00:46:05
- Community feedback: 8037/8038 most controversial due to complexity concerns00:55:02
testing progress
client updates
Action Items
- Maria Silva, client teams: Review 7904 precompile tests for caching issues; verify EC_RECOVER/EC_PAIRING inputs vary00:17:04
- NetherMind (Camille), Monad: Client teams: Share internal benchmarks contradicting precompile cost decreases00:22:02
- Spencer: Analyze if 8037 reservoir mechanism introduces new worst-case scenarios for benchmarks00:48:29
- Butta, Maria Silva: Follow up with concerned entities on 8037 dynamic cost complexity; quantify actual impact00:59:06